[00:49:03] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Degraded RAID on analytics1055 - https://phabricator.wikimedia.org/T172809#3510136 (10Cmjohnson) The new disk arrived today and will be swapped on 08/30. [01:23:09] (03CR) 10Ottomata: "Couple of comments, BUT THIS IS TOTALLY AWESOOOME!" (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) (owner: 10Joal) [03:46:41] 10Analytics, 10MediaWiki-API, 10Patch-For-Review: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables - https://phabricator.wikimedia.org/T137321#3564821 (10bd808) [03:47:11] 10Analytics, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: ApiAction log in data lake doesn't record Wikibase API actions - https://phabricator.wikimedia.org/T174474#3564823 (10bd808) See the comment at the bottom of https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/ApiAction: ``` --... [06:38:07] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: kafka-jumbo.cfg partman recipe creation/troubleshooting - https://phabricator.wikimedia.org/T174457#3564922 (10elukey) First thing that came up in my mind is a previous fight with Partman that ended up in this note: https://wikitec... [07:22:05] 10Analytics, 10Operations, 10User-Elukey, 10User-Joe: rack/setup/install conf1004-conf1006 - https://phabricator.wikimedia.org/T166081#3564969 (10elukey) [07:22:57] 10Analytics, 10Operations, 10User-Elukey, 10User-Joe: rack/setup/install conf1004-conf1006 - https://phabricator.wikimedia.org/T166081#3284834 (10elukey) Due to the new Kafka Jumbo cluster (and other things like the Eventlogging cleaner script) I didn't get much time to schedule/plan this work, that may en... [09:01:55] joal: o/ [09:02:09] I am going to restart druid jvm daemons on druid1001 [09:05:48] aaaaand oozie complained [09:05:51] of course [09:07:01] !log restart pageview-druid-hourly-wf-2017-8-30-7 in Hue after druid1001 daemons restart [09:07:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:07:14] !log restart all jvm daemons on druid100[123] for security updates [09:07:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:17:56] 10Analytics-Kanban, 10DBA, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3565122 (10Marostegui) >>! In T170952#3562297, @Ottomata wrote: > Hm, for now I think this is good, so that the slaves match each other. If we want everything to be a... [09:42:20] Hi elukey ! Sorry I missed your ping - Thanks for the restarts :) [09:47:26] joal: no problem! I am fighting with partman and kafka-jumb [09:47:39] my day is full of joy [09:47:50] if you are ok I'd update druid100[23] too [09:48:19] elukey: please go ahead with druid100[23456] [09:48:33] are 456 already installed? [09:48:40] I might have missed some updates [09:48:56] elukey: And about partman, I'm behind you, but5 I hide :) [09:48:58] ah they are! [09:49:02] elukey: yes they are :) [09:57:20] 10Analytics, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: ApiAction log in data lake doesn't record Wikibase API actions - https://phabricator.wikimedia.org/T174474#3565190 (10Ladsgroup) @bd808 Thanks for the explanation. The gerrit patch seems to be staying there for a very long time, is there any... [11:25:16] Hi those of the ATeam that are currently awake! Do you mind If I create a https://wikitech.wikimedia.org/wiki/Analytics/WMDE page on Wikitech to document our things? :) [11:25:31] 10Quarry: CHAR_LENGTH does not return the character count - https://phabricator.wikimedia.org/T174543#3565363 (10WhitePhosphorus) [11:25:34] 10Analytics-Kanban, 10DBA, 10Patch-For-Review: Inconsistent default charset for analytics slaves - https://phabricator.wikimedia.org/T170952#3565375 (10Marostegui) 05Open>03Resolved Closing this as per my chat with Luca Thank you! [11:25:38] Hi addshore :) [11:25:46] addshore: no problem for me :) [11:26:12] +1 :D [11:27:03] 10Analytics, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: ApiAction log in data lake doesn't record Wikibase API actions - https://phabricator.wikimedia.org/T174474#3563202 (10Addshore) If you need someone one off however it will still be in the raw actionapi logs if I am not mistaken! [11:31:30] * elukey lunch! [11:31:43] addshore: +1 [11:32:00] 1+1=2 :D +2 {{doing}} [11:35:40] (03CR) 10Joal: [C: 031] "Looks god to me ! Please merge when you want." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/374623 (https://phabricator.wikimedia.org/T174484) (owner: 10Mforns) [12:00:24] (03PS9) 10Joal: Add tranquility to the banner streaming job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) [12:00:56] (03PS10) 10Joal: Add tranquility to the banner streaming job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) [12:02:50] (03PS11) 10Joal: Add tranquility to the banner streaming job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) [12:18:27] lucaaaa, sorryyy... pushing code right now [12:18:46] mforns: don't worry! I am fighting with partman today :) [12:18:56] batman? [12:18:58] feel free to work on it, we can sync later on ok? [12:19:02] ok [12:48:59] (03PS12) 10Joal: Add tranquility to the banner streaming job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) [12:51:07] (03CR) 10jerkins-bot: [V: 04-1] Add tranquility to the banner streaming job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) (owner: 10Joal) [12:52:19] mforns: ah also happy anniversary! [12:52:22] :) [12:52:37] :D thanks [12:53:18] ¡Feliz cumpleaños! [12:54:54] A-team, I'll move my usual break later in the day today to have some time with Lino - Will miss standup but will send e-scrum [12:55:21] gracias joal :] [12:55:29] ok [13:19:43] (03PS13) 10Joal: Add tranquility to the banner streaming job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) [13:26:23] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: kafka-jumbo.cfg partman recipe creation/troubleshooting - https://phabricator.wikimedia.org/T174457#3565634 (10elukey) The following recipe is almost working fine, except that I didn't manage to create the srv logical volume yet for... [13:42:30] ottomata: [13:42:35] root@kafka-jumbo1001:~# pvs PV VG Fmt Attr PSize PFree /dev/sda3 vg-flex lvm2 a-- 893.46g 0 /dev/sdb1 vg-data lvm2 a-- 21.83t 0 [13:42:38] root@kafka-jumbo1001:~# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert srv vg-data -wi-ao---- 21.83t placeholder vg-flex -wi-a----- 856.21g root vg-flex -wi-ao---- 37.25g [13:42:43] sigh, horrible paste [13:45:42] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: kafka-jumbo.cfg partman recipe creation/troubleshooting - https://phabricator.wikimedia.org/T174457#3565720 (10elukey) Forgot a "\" after a ".", there where is left to the reader as exercise :D This is what we end up with: ``` roo... [13:47:13] ottomata: I have modified substantially the code for spark/trnquility if you want to have a look [13:47:22] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: kafka-jumbo.cfg partman recipe creation/troubleshooting - https://phabricator.wikimedia.org/T174457#3565738 (10elukey) The /boot partition might need more space, I used the `prometheus.cfg` configuration as baseline and forgot to up... [13:48:05] joal: since last night? [13:48:06] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: kafka-jumbo.cfg partman recipe creation/troubleshooting - https://phabricator.wikimedia.org/T174457#3565739 (10Ottomata) Weird! /dev/dm-0 is a LVM partition? [13:48:07] did you see my commetns? [13:49:30] 10Analytics-EventLogging, 10Analytics-Kanban, 10AbuseFilter, 10CirrusSearch, and 30 others: Possible WMF deployed extension PHP 7 issues - https://phabricator.wikimedia.org/T173850#3541977 (10Ottomata) I fixed the two warnings from the EventLogging extension, but I have not been able to test them. If anyo... [13:49:54] I saw your comments ottomata, and actually modified a lot, assuming that we'd reuse the tranquility singleton [13:51:51] ok looking! [13:52:47] oh nice! [13:52:57] joal i was thinking about something like this, but thought for this experiement keeping it all in the one object was fine for now [13:52:59] but this is a good idea [13:53:04] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: kafka-jumbo.cfg partman recipe creation/troubleshooting - https://phabricator.wikimedia.org/T174457#3565761 (10elukey) I can definitely see the following that looks correct: ``` root@kafka-jumbo1001:/# mount | grep root /dev/mapper... [13:53:54] mforns: my nightmare with partman is probably done for today, if you want we can chat about EL (or do it after standup if you are working on it) [13:54:01] (grabbing a coffee in the meantime) [13:54:26] elukey, I'm working on it, but we can chat in da caiv [13:55:51] mforns: as you prefer! [13:56:04] I'm in the batcave :] [13:56:28] getting a coffee and then joining [13:56:33] I have dibs on batman [13:57:28] oh nice joal, so this means you can use tranquility without having to create a case class for your data? because you just use a Map? [13:57:42] or, no [13:57:42] IndexedSeq [13:57:43] ? [13:58:14] ottomata: You should use a Map, and provide dimensions and timestamp as named in Map [14:00:08] i see your RDD is a buncha maps [14:00:16] and the job gets the list of dimensions to extract from the map [14:00:18] nice [14:01:08] the other things to configure is timestamp related and aggregsation related [14:01:25] right [14:01:26] cool [14:01:46] joal: , what do you think about my comment about producing to kafka, then consuming again to produce to druid? [14:01:57] probably not necessary now, but could make things more resilient? [14:02:19] ottomata: I don't think it would [14:02:29] might make more streaming things flexible later, e.g. if all spark intermediate streams are in kafka topics, then we could use other things to consume them too [14:02:31] e.g. KSQL :) [14:02:47] hehe :) [14:02:52] But, more IOs [14:03:04] yeah, but that's the whole premise of kafka streams stuff anyway [14:03:09] kafka is the intermediate for all streams [14:03:10] IO is ok [14:03:15] trade of for fledibility [14:03:18] anyway, i don't think we should do this now [14:03:22] but it is an idea to keep in mind [14:03:30] sure ottomata [14:03:41] the IOs here are pretty low anyway, right? [14:03:45] webrequest only comes in once [14:03:48] mforns: can't hear you in the cave, or you are afk :D [14:03:49] its just the aggregates that get emitted [14:04:11] (03CR) 10Ottomata: "Very very very nice! One comment, feel free to ignore." (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) (owner: 10Joal) [14:04:13] (03CR) 10Ottomata: [C: 031] Add tranquility to the banner streaming job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) (owner: 10Joal) [14:04:16] 10Analytics-Kanban, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3360724 (10phuedx) @Marostegui: In T172291#3564966, you reported that available space on dbstore1002 had increased to 91%. That increase took a little over 16 hours. Do we know what caused it? [14:06:17] 10Analytics-Kanban, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3565857 (10Marostegui) >>! In T168303#3565851, @phuedx wrote: > @Marostegui: In T172291#3564966, you reported that available space on dbstore1002 had increased to 91%. That increase took a little over 16... [14:06:36] no jmxtrans on druid100[456]? [14:07:19] !log restart java daemons on druid100[456] for jvm security updates [14:07:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:31:32] elukey: ?? [14:31:47] is jmxtrans not puppetized? they are all including the same thigns [15:01:27] a-team get up, stand up [15:02:41] fdans: , joal? [15:12:25] 10Analytics, 10Wikimedia-Stream, 10Patch-For-Review: Create /v2/schema/:schema_uri endpoint for eventstreams that proxies schemas from eventbus - https://phabricator.wikimedia.org/T160748#3566131 (10Ottomata) [15:14:52] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Degraded RAID on analytics1055 - https://phabricator.wikimedia.org/T172809#3566159 (10Cmjohnson) @elukey I swapped the disk in slot 1...megacli still shows failed but also does not show the updated s/n for the new disk it could be preserved cache. Please try an... [15:32:15] elukey, whenever you have time [15:43:39] mforns: I have 15mins now, but I can check the code review [15:45:15] ok, just added a missing method [15:45:19] 10Analytics-Cluster, 10Analytics-Kanban: Hadoop cluster expansion. Add Nodes - https://phabricator.wikimedia.org/T152713#3566380 (10RobH) [15:45:59] RECOVERY - Hadoop DataNode on analytics1055 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [15:46:38] RECOVERY - Hadoop NodeManager on analytics1055 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [15:47:57] ottomata: the device names got screwed up but from df -h it seems that an1055 is good [15:52:02] gr8 :) [15:52:35] hadoop-hdfs-datanode-analytics1055.log's log looks weird, checking it our [15:53:34] mmm maybe it needed only a puppet run first [15:54:47] !log re-added analytics1055 among the hdfs/yarn worker after maintenance [15:54:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:09:37] 10Analytics: Request for python package csvsort on stat1005.eqiad.wmnet - https://phabricator.wikimedia.org/T174577#3566546 (10faidon) [16:22:28] PROBLEM - HDFS corrupt blocks on analytics1001 is CRITICAL: CRITICAL: 75.86% of data above the critical threshold [5.0] [16:23:00] checking --^ [16:23:30] so the last corrupt blocks event auto-resolved https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?panelId=39&fullscreen&orgId=1 [16:23:39] but this one is probably due to analytics1055's new disk [16:24:24] since it is syncing from other replicas all the missing data [16:24:24] that makes sense, now that is coming back up [16:24:26] ya [16:24:40] all right [16:24:44] good that these alarms works :) [16:25:40] mforns: I am free now! (finally) [16:25:50] elukey, to the cave! [16:26:27] elukey, batcave-2 [16:41:55] 10Analytics-Kanban, 10Operations, 10ops-eqiad: Degraded RAID on analytics1055 - https://phabricator.wikimedia.org/T172809#3566690 (10elukey) 05Open>03Resolved Done! Host back to working, thanks Chris! [16:45:34] 10Analytics-EventLogging, 10Analytics-Kanban, 10AbuseFilter, 10CirrusSearch, and 30 others: Possible WMF deployed extension PHP 7 issues - https://phabricator.wikimedia.org/T173850#3566716 (10Addshore) Regarding Cognate: ``` File: /home/reedy/git/mediawiki/core/extensions/Cognate/src/CognateHooks.php > Li... [16:47:29] 10Analytics-EventLogging, 10Analytics-Kanban, 10AbuseFilter, 10CirrusSearch, and 30 others: Possible WMF deployed extension PHP 7 issues - https://phabricator.wikimedia.org/T173850#3566721 (10Reedy) Seems to come from.. https://github.com/sstalle/php7cc/blob/67f3586e4adadc79f0625d1de5a935ad0ed99178/src/Nod... [17:01:24] * elukey off! [17:27:15] afk for a bit... [18:22:17] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3567272 (10Pchelolo) [18:22:21] 10Analytics, 10ChangeProp, 10EventBus, 10MW-1.30-release-notes (WMF-deploy-2017-07-25_(1.30.0-wmf.11)), and 3 others: Support posting Jobs to EventBus simultaneously with normal job processing - https://phabricator.wikimedia.org/T163380#3567270 (10Pchelolo) 05Open>03Resolved All the jobs are being post... [18:26:49] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Set up LVS and VirtualHost for RunSingleJob.php - https://phabricator.wikimedia.org/T174599#3567282 (10Pchelolo) [18:27:22] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Set up LVS and VirtualHost for RunSingleJob.php - https://phabricator.wikimedia.org/T174599#3567297 (10Pchelolo) a:05Pchelolo>03Joe [18:30:06] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Add the ability to sign and verify jobs - https://phabricator.wikimedia.org/T174600#3567300 (10Pchelolo) [18:42:41] (03CR) 10Mforns: [V: 031 C: 031] "Looks awesome!" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/373373 (owner: 10Nettrom) [18:51:00] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Add the ability to sign and verify jobs - https://phabricator.wikimedia.org/T174600#3567377 (10Ottomata) Not sure if this is relevant, but I'm working on a generic key & cert generation tool over in T166167. Could be useful for gen... [19:02:44] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Write generic certificate management software for use with Puppet and Self Signing CAs. - https://phabricator.wikimedia.org/T166167#3567433 (10Ottomata) FYI, etherpad overview and discussion here: https://etherpad.wikimedia.org/p/certpy [19:09:38] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3567468 (10mforns) Hi all! While following up on EL performance regarding Popups experiment, I looked at the mysql consumer logs and I think EL is receiving around 200 events pe... [19:16:58] ottomata: still here ? [19:17:29] ottomata: looked at your comment on options, and only relevant thing I found on the internetz was that" https://github.com/scopt/scopt/issues/39 [19:17:42] ha ya [19:17:52] ottomata: If you're ok, I'll leave the option parsing generic aspect for later :) [19:18:37] joal, totall ok witth that, that looks complicated [19:18:39] alternatively [19:18:51] you could just parse args twice differently? [19:19:01] or do a --tranquility-options="..." [19:19:09] and parse that with the Tranquility class opt parser [19:19:16] to get your TranquilityConf case class [19:19:17] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3567505 (10Tbayer) @mforns The estimate was for around 100 events/second //on average//, and the new sampling rate was chosen based on the event rate from the previous instrume... [19:19:19] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: Add the ability to sign and verify jobs - https://phabricator.wikimedia.org/T174600#3567507 (10GWicke) Signed JSON blobs are kind of what [JWTs](https://en.wikipedia.org/wiki/JSON_Web_Token) are designed for. There are good librarie... [19:19:28] but, ja, am totally fine with doing that later' [19:19:36] cool :) [19:21:23] 10Analytics-Kanban, 10User-Elukey: Calculate how much Popups events EL databases can host - https://phabricator.wikimedia.org/T172322#3567516 (10mforns) @Tbayer I see, makes total sense. Thanks for the explanation! [19:26:17] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Try to make tranquility work with Spark - https://phabricator.wikimedia.org/T168550#3368133 (10JAllemandou) [19:26:35] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Make tranquility work with Spark - https://phabricator.wikimedia.org/T168550#3368133 (10JAllemandou) [19:28:01] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Make tranquility work with Spark - https://phabricator.wikimedia.org/T168550#3567562 (10JAllemandou) [19:32:03] Hi nuria_, are you still here? [19:35:39] (03CR) 10Nettrom: [C: 04-1] "> Looks awesome!" (033 comments) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/373373 (owner: 10Nettrom) [19:52:11] 10Analytics, 10cloud-services-team: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation - https://phabricator.wikimedia.org/T166712#3305259 (10Krenair) Ping @Nuria [20:02:22] RECOVERY - HDFS corrupt blocks on analytics1001 is OK: OK: Less than 60.00% above the threshold [2.0] [20:04:02] (03CR) 10Mforns: [V: 031 C: 031] "I was creating the puppet patch to trigger your reportupdater jobs, and I found that the convention for reportupdater query directories na" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/373373 (owner: 10Nettrom) [20:12:50] (03PS3) 10Nettrom: Add page creation configuration and queries [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/373373 [20:15:32] joal: did anyone do a larger estimate of the impact of this bugfix for monthly per-domain uniques? https://phabricator.wikimedia.org/T143928#3301667 [20:15:51] (03CR) 10Nettrom: "I renamed the directory to "page-creation" per comment about directory naming conventions above. Also fixed the autopatrolled count. Argua" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/373373 (owner: 10Nettrom) [20:16:17] i saw your note there that it changed the offset by about ~10% (was that about daily or monthly?) [20:16:39] HaeB: It was daily - we didn't check for monthly because of the computation price [20:17:05] ok then i'll run an analogue of the monthly job with the old definition [20:17:13] HaeB: Thanks :) [20:17:20] understood, but that's the number we have been tracking as a core metric for a long time ;) [20:18:18] No problem HaeB, I understand as well :) By the way, this request would be a good example of one to be tried in nice queue :) [20:18:23] also, i did some sanity checks on the new family version some weeks ago (before wikimania), but still need to follow up further on the ticket [20:19:07] HaeB: I assume that there is nothing completely wrong or we would already have heard from you then :) [20:19:20] agreed (i already noticed the rollout ;) https://phabricator.wikimedia.org/T156841#3563330 ( [20:19:30] hopefully it's not too slow [20:20:40] HaeB: It'll slow it a bit, but shouldn't be very noticeable (as long as no other user takes the whole cluster for a long preiod) [20:25:48] joal: well, the monthly WP numbers are in line with the previous estimates, but i still want to run some more checks [20:26:44] ... in particular, check that for all project families, countries and months/days, unique_devices_project_wide_monthly.uniques_estimate >= MAX(unique_devices_per_domain_monthly.uniques_estimate) (taken over all language versions) [20:27:44] i guess there might be some smaller countries and families there could be outliers where this is violated, but in general it should hold [20:27:56] did you do any checks of this sort already? [20:28:54] (03CR) 10Mforns: [V: 032 C: 032] "LGTM! Merging :]" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/373373 (owner: 10Nettrom) [20:29:27] oh and there was a general issue regarding some smaller sister projects (not affecting the WP numbers) -will follow up on phab [20:29:28] HaeB: I can't recall - I don't think so, but I'm not sure [20:44:34] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3567920 (10mforns) @Nettrom I merged the queries patch. When the puppet patch is reviewed and merged, you should see your report... [21:12:36] off for tonight a-team, see y'all [21:12:46] byeeeeee [21:13:15] by joal - i'll have some more UD questions later ;) [21:13:34] byEEEE *cough* ;) [21:16:09] laaters! [21:35:55] msg joal yesss [22:12:50] laters all! [22:42:52] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, 10Services (doing): Generalized rate limiting, deduplication, and job scheduling module - https://phabricator.wikimedia.org/T173447#3568473 (10Pchelolo) [22:42:54] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3568474 (10Pchelolo) [22:42:59] 10Analytics, 10ChangeProp, 10EventBus, 10Services (later): Generalise deduplication in ChangeProp - https://phabricator.wikimedia.org/T157090#3568471 (10Pchelolo) 05Open>03Resolved We have redid-based reduplication running in production now that's exactly matching the algorithm used in the JobQueue. Th... [23:57:37] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: Make webperf eventlogging consumers use eventlogging on Kafka - https://phabricator.wikimedia.org/T110903#1589603 (10thcipriani) For clarification, can this task be removed as a blocker for sunsetting salt? I saw in T118772#352...