[07:17:34] Hi team - Lino is sick today, I'll be on/off at irregular times [07:39:54] joal: ack! bonjour! [08:07:00] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10User-Elukey: Upgrade to Superset 0.35.2 - https://phabricator.wikimedia.org/T242870 (10elukey) Checked a lot of charts and everything seems rendering fine, all good from my side! @Nuria do you prefer to do the last round of tests/checks befo... [08:10:00] Hi elukey - I'm gonna correct the CR for labstore rysnc - Maybe I can add an houly run, to see quicklu if it works or not? [08:11:32] joal: I can force it manually when ready [08:11:40] systemctl restart etc... [08:11:49] (magic of timers!) [08:11:56] elukey: great :) [08:12:08] * joal is afraid of magic, but trust elukey :) [08:20:56] the bigtop people answered! http://mail-archives.apache.org/mod_mbox/bigtop-user/202001.mbox/browser [08:21:01] really interesting [08:21:22] yesterday I thought I had also sent an email to user@spark.etc.. but never happened [08:21:25] mmmm [08:25:18] just re-sent it [08:25:30] https://lists.apache.org/thread.html/r21d225dd522c6940430e33abc63ec1935a42b26de593d48fbb90e99c%40%3Cuser.spark.apache.org%3E [08:25:44] interesting answers elukey [08:25:47] joal: this is for the encryption weirdness --^ [08:25:52] up [08:25:54] yup [08:43:48] elukey: how does that new patch version look? [08:46:01] running puppet compiler [08:50:30] joal: not sure if I am missing something, but the exec start's bash command renders as [08:50:32] /bin/bash -c '/usr/local/bin/hdfs-rsync --dry-run -r -t --delete --chmod=go-w hdfs:///wmf/data/archive/mediawiki/history/{$(/bin/date --date=\"$(/bin/date +%Y-%m-15) -1 month\" +\"%Y-%m\"),$(/bin/date --date=\"$(/bin/date +%Y-%m-15) -1 month\" +\"%Y-%m\")} file:///srv/dumps/xmldatadumps/public/other/mediawiki_history/' [08:51:18] (without dry-run) [08:51:24] elukey: Ah [08:51:25] :) [08:51:33] I wondered and was triple-checking :) [08:51:55] no exclude? [08:52:28] not that I see [08:52:31] elukey: Ahhhhh - I think I know why [08:53:03] also I removed the "\" and tried to run it from stat1004, I get this (not sure if it is due to me being stupid or not) [08:53:07] elukey: correcting about the exclude [08:53:09] 2020-01-16T08:51:41.265 INFO HdfsRsyncExec CREATE_DIR [dryrun] - file:/srv/dumps/xmldatadumps/public/other/mediawiki_history/2019-12 [08:53:12] 2020-01-16T08:51:41.570 INFO HdfsRsyncExec CREATE_DIR [dryrun] - file:/srv/dumps/xmldatadumps/public/other/mediawiki_history/2019-12/aawiki [08:53:15] Exception in thread "main" java.lang.IllegalStateException: SRC_CONFLICT - Trying to copy multiple objects with the same filename at the same destination [08:54:33] makes sense elukey - bug: two time "-1 month" (instead of -2 the second one [08:56:20] ah okok [08:56:45] elukey: I don't understand why the " are escaped - I escape them for puppet, expected they'd be rendered as " only [08:58:35] elukey: trying a new version [08:59:16] joal: I always confuse how puppet does things, let's keep them in this way for the moment [08:59:42] elukey: I changed the variable definition to use '' and removed escaping from " [08:59:50] elukey: see new patch [09:02:23] joal: please don't kill me but if you do 'file://${miscdatasetsdir}/mediawiki_history/' it will be verbatim, no variable subs [09:02:36] Ahhhhh [09:02:44] I'm such a puppet dumb [09:03:24] but what about escaped " then ... [09:03:29] Will keep them escaped for now [09:03:33] yes exactly [09:03:43] I think that the puppet compiler's output might trick us now [09:03:54] let's keep that fixed and change if needed [09:04:14] yup [09:04:16] sending patch [09:12:28] elukey: patch ready :) [09:17:57] joal: the exclude is now added everywhere https://puppet-compiler.wmflabs.org/compiler1003/20379/labstore1006.wikimedia.org/ [09:18:54] fdans: when you have a moment let's discuss https://phabricator.wikimedia.org/T237752 [09:29:16] elukey: isn't that what we wanted? [09:30:26] also elukey, I don't see the rendering for the mediawiki_history_dumps :( [09:33:00] joal: I thought we wanted to modify only one rsync, but possibly I am loosing track of the goals [09:33:04] fine for me, just wanted to check [09:33:32] the ExecStart for history is in the change catalog, lemme pull it out [09:34:07] elukey: I womder of the current version of all the jobs include the exclude (given how it is configured) [09:34:28] elukey: let's batcave for a minute if ok - It'll be easier :) [09:36:37] one sec, doorbell! [09:38:38] joal: I am in! [09:38:48] joining ! [10:03:17] elukey: sorry I totally missed your ping while I switched places [10:03:23] i'm here whenever [10:11:48] (03CR) 10Fdans: "nuria: my bad, I didn't fix the dev config after breaking it with this change, Will follow up with another patch set fixing it." (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) (owner: 10Fdans) [10:16:58] fdans: no problem! I was only leaving a ping for when you had a moment :) [10:17:18] the redirect to /v2 option is very simple and appealing [10:20:43] elukey: to me it's awkward to have the url change like that when entering stats.wikimedia.org [10:21:32] plus /v2/ was never meant to be a form of version control. /v2/ could have been /new/, /beta/, /alpha/ ... it was simply a different namespace to store the new version while it was under development [10:23:02] I don't think it is ackward, seems easy enough to get, but you guys decide :) [10:23:25] Even Timo mentioned that the redirect to /v2 was nicer, I think that people already consider it as versioning [10:23:29] despite our efforts [10:36:18] 10Analytics, 10serviceops, 10Product-Analytics (Kanban): Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10akosiaris) > Our new Helm chart templates were not originally developed to handle multi-service deployment charts On the c... [11:19:01] 10Analytics, 10Analytics-SWAP, 10GLOW: Viewing Santali and Javanese characters on SWAP via Chrome only displays Tofu signs - https://phabricator.wikimedia.org/T242490 (10Bugreporter) [11:42:17] * elukey lunch! [12:57:33] For anyone from wmde-team - There is a heavy hive query ongoing on the cluster under analytics-privatedata user - This could for some optimization to prevent using that many resource - It would be great for the person running it to get in touch please :) [12:59:51] joal: I can find who started it, do you know when more or less it was submitted? [13:00:20] elukey: You were supposed to be gone for lunch :) [13:00:32] ah now I see [13:00:39] well I was more than an hour ago :D [13:00:44] Started Thu Jan 16 12:48:46 UTC 2020 [13:01:08] we log in syslog usage of kerberos-run-command [13:01:22] so in theory I should be able to find who launched it [13:01:25] nice elukey :) [13:03:22] the winner is Goran :) [13:03:39] joal: --^ [13:03:44] from stat1004 [13:03:52] sudo cumin 'stat*' 'grep analytics-privatedata /var/log/syslog' [13:05:32] I think this is launched from an automated script - I'll send an email asking for improvements [13:05:51] joal: please Cc: internal [13:05:55] ack elukey [13:07:50] πŸ™„ [13:31:05] hello GoranSM :) [13:35:28] elukey: how is our copy going? [13:35:56] joal: I think it was done a while ago [13:36:01] \o/ [13:36:22] and I restarted it again, didn't see anything logged [13:36:30] so it should be working just fine :) [13:36:40] Hurray :) [13:37:03] elukey: do you want us to late it bake a bit before modifying others? [13:37:05] congrats joal ! [13:37:31] elukey: I don't think it would be very usefull as it only do something once a month ) [13:41:00] Ah elukey - Thanks for patching my missing double-$ [13:41:02] :S [13:46:20] joal: let's modify another one that runs more frequently, and on Monday we'll flip all [13:46:23] is it ok? [13:48:49] very ok elukey :) [13:49:35] elukey: mediacounts? [13:53:07] joal: +1 [14:04:18] joal: hello [14:04:34] Hi nuria [14:05:20] joal: i got numbers from select and for a month number is 48,864,420 of commons files used internally, this still seems a big high [14:06:00] joal: as it is about 80% [14:06:03] nuria: it feels high - I realized there are 2 things we should have filtered for: http_status (200 or 304) [14:06:23] joal: the mediarequests table only has images , not http codes [14:06:38] Ah true ! [14:06:46] I played with werequest a bit [14:07:11] joal: and teh second thing? [14:07:13] *the [14:07:59] The other thing I noticed is: when using VisualEditor and inserting an image, you search commons - You hit upload to display results, but all of those are not included in the page (and possibly not included in any page) [14:08:39] This one is difficult to filter out as many wikis only send the main host as referer, not details [14:10:33] ah ya [14:13:02] elukey: what's the process for getting added to the gpu-testers group on stat1005? i have a tensorflow model that is taking over a few hours to train on stat1007 CPUs and I suspect that would go way down on GPUs and let me actually do some tuning in an efficient manner [14:13:58] nuria: joal the mediarequests table already excludes http 304 https://github.com/wikimedia/analytics-refinery/blob/master/oozie/mediarequest/hourly/mediarequest_hourly.hql#L52 [14:14:24] fdans: right! [14:14:54] isaacj: 5 euros to me usually [14:14:54] (as did mediacounts) [14:14:57] :D [14:15:02] does that come with a pool pass? [14:15:08] of course! [14:15:11] isaacj: hell no [14:15:14] no pool! [14:15:18] :D [14:15:20] well i'm in -- let me just set up a bitcoin account :) [14:15:32] fdans: you strike a hard bargain [14:15:49] isaacj: I represent elukey 's interests here [14:15:50] joal: and did you run again your spark job? [14:16:06] classic analytics shakedown [14:16:12] nuria: could you please approve/nack https://phabricator.wikimedia.org/T241838 for this Hadoop access? [14:16:39] POOL PASS jajaja [14:16:53] nuria: I did yesterday, gist is updated with new numbers (I sent a message yesterday on chan, but it was very busy) [14:18:47] nuria: can you review/+1 https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565280/ to allow isaacj to use the GPU? [14:19:20] thanks both! [14:19:49] isaacj: the plan is eventually to allow everybody, but for now I wanted to keep testing for people that really needed to avoid conflicts etc.. [14:20:53] joal: ya, numbers differ quite a bit [14:21:01] indeed !! [14:21:34] nuria: I can't imagine how to get more precise numbers [14:21:49] nuria: also, we should remember those are for images only [14:22:12] isaacj: you are free to use the GPU on stat1005 [14:22:36] joal: the ones for teh spark job, ya [14:22:37] isaacj: also please check https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/AMD_GPU [14:22:43] yes nuria [14:22:45] joal: but still it does not explain teh difference [14:23:15] nuria: I really think you can't say that traffic is a good proxy for images used in pages [14:23:26] joal: ya, agree [14:23:52] nuria: particularly when doing a count distinct and not looking at cutting the long-tail [14:24:15] nuria: I can't say my numbers are good, but I think they are closer to reality than the ones you get from traffic [14:24:26] joal: why would cutting the long tail make the number more precise? [14:25:41] nuria: I wouldn't say more precise, cause cutting the long tail is actually very difficult - What I mean is trying to remove hits that comes from artifacts (search for instance) [14:26:08] joal: and did you leave vide out for any reason? [14:26:17] joal: from your spark job [14:26:27] nuria: a fun way to try to get only "pageview-related" content is to play with fingerprinting and timestamps between pageviews and images [14:26:52] joal: ya, this is a project that would take more than 1 day [14:26:59] elukey: thanks! looks like tensorflow is seeing it now so should be all good [14:27:08] nuria: I use images only cause that was what we discussed - "images" - I didn't think of growing the set to rich-media (sounds, video) [14:27:26] isaacj: did you see the note about tensorflow 1.14.1 ? [14:27:29] nuria: it could be done relatively easily, even with some categorization [14:28:05] joal: let's do that just to have those numbers in case somebody asks, i do not think categorization is needed [14:28:25] yeah, i was hoping that I didn't need to be in any special groups anymore so i went through the installation stuff already :) thanks for including that note in there -- was easy to get it setup once i found that [14:28:31] nuria: I'll do the categorization, it's included in the dev price ;) [14:28:37] No pool pass though :) [14:31:02] nuria: I'm planning on using the extension list from mediarequest identification code - ok for you ? [14:32:02] elukey: 5-10x speedup. thanks!! [14:32:56] isaacj: wooooowwww [14:33:08] joal: sure, i think that is the totality [14:33:15] ok great :) [14:33:24] isaacj: those 5 eruros were a good investment [14:33:48] yep! and it's very exciting because i did literally nothing but switch machines :) [14:34:28] isaacj: one question for you - the next versions of the amd gpu drivers/tools/etc.. is coupled with tensorflow 2.0, and Miriam asked to me to wait a bit before upgrading.. what about your use cases? [14:34:36] (trying to gather some info) [14:36:46] nuria: something else I forgot in the existing code: ignore-case :( [14:36:49] Will do it now [14:36:54] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10User-Elukey: Upgrade to Superset 0.35.2 - https://phabricator.wikimedia.org/T242870 (10Nuria) No, i think @mforns work is sufficient [14:37:12] elukey: i'm actually working with Keras (https://keras.io/), which uses Tensorflow as its backend. my models are pretty standard (nothing too custom) so 1.14 seems to be working fine. some internet searching suggests 2.0 is well-supported too so I don't expect it would cause any issues for me. I'm fine to wait till Miriam is ready though [14:37:56] isaacj: yes yes Miriam is currently the GPU overlord, I can't do anything without her approval [14:37:59] :D [14:38:06] :) [14:38:57] 10Analytics, 10serviceops, 10Product-Analytics (Kanban): Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) > Why mess with wmf.releasename Because wmf.releasename doesn't currently consider the service's name, only the... [14:43:34] nuria / joal: mediarequests filter webrequests to only 200 and 206: https://github.com/wikimedia/analytics-refinery/blob/master/oozie/mediarequest/hourly/mediarequest_hourly.hql#L52 [14:43:45] ack milimetric :) [14:45:01] doh, fran already said that, sorry [14:45:07] stupid time dimension [14:46:03] joal: does it look good ? https://puppet-compiler.wmflabs.org/compiler1001/20393/labstore1006.wikimedia.org/ [14:46:24] looking elukey [14:47:19] it does elukey :) [14:51:52] nuria: if you are ok then I'd deploy superset [14:52:30] joal: it is running now on 1006 [14:52:38] \o/ [14:52:59] elukey: let's monitor that - I'm confident but will still feel better once confirmed :) [14:53:07] elukey: I'm sorry you have to do it though [14:53:32] it already completed without any logs or exceptions etc.. [14:53:55] ok, let's confirm new data comes up :) [14:54:38] Ah elukey - daily files only - I should have picked pageview :) [15:00:58] 10Analytics, 10Analytics-Kanban, 10Release Pipeline, 10Patch-For-Review, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) Bump on this review please! I'd love to deploy this asap (maybe even before all hands) to start testing in k8s. We... [15:06:34] joal: I just replaced cdh packages with bigtop ones on analytics1031 in the test cluster (a worker) [15:06:53] elukey: This is fun!!! [15:06:53] all that we need is there, hdfs datanode and journal works but yarn does not [15:07:01] ok [15:07:03] I think due to https://issues.apache.org/jira/browse/YARN-8310 [15:07:29] but it looks promising, maybe a stop-of-the-world upgrade can work [15:07:47] That would be so AWESOME :D [15:08:11] elukey: how shall we handle access requests for new people gaining Hadoop access (so far we've mostly been backfilling existing Hadoop users with a Kerberos account), as part of the initial ticket or should the SRE merging the request simply enable it along? [15:08:29] 10Quarry, 10Patch-Needs-Improvement: Remember filters: "All queries", "Published queries", etc. chosen by user in recent queries page - https://phabricator.wikimedia.org/T76084 (10Aklapper) [15:09:08] moritzm: I think we can do it as part of the ticket if the user requests to be in the privatedata user group, what do you think? [15:10:31] joal: also the nice thing is that up now I have used all the pre-existing configs [15:10:37] no changes except from apt repos [15:10:53] yeah, I think that makes sense. can you send a mail to ops-private list to let people know? [15:11:00] this is also very very cool ! [15:11:00] sure! [15:11:55] (03CR) 10Elukey: [V: 03+2 C: 03+2] Release Superset 0.35.2 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/565037 (owner: 10Elukey) [15:14:30] !log stop superset as prep step for upgrade [15:14:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:17:23] !log upgrade superset to 0.35.2 [15:17:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:18:52] team - Going for kids - Also, Lino being well I'm gonna help this evening and miss standup - e-scrum is on its way, I'll be back after they are in bed [15:20:49] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10User-Elukey: Upgrade to Superset 0.35.2 - https://phabricator.wikimedia.org/T242870 (10elukey) @mforns superset upgraded! Can you check superset.wikimedia.org to see if you spot any anomaly? [15:20:59] ack! [15:21:03] superset upgraded! [15:29:09] be back in a bit! [15:30:49] thanks mforns! [15:41:25] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10User-Elukey: Upgrade to Superset 0.35.2 - https://phabricator.wikimedia.org/T242870 (10Nuria) ping @cchen (which i think is the heaviest user ) that upgrade has happened [15:45:58] joal: let's talk when you are back about the time ranges issues with bots 24 hours computation, me slow, i still do not get how to do that efficiently [15:49:16] elukey: looked arround a bit in superset , looks good so far [15:51:39] nuria: thanks! [15:57:03] (03PS1) 10Mforns: Add ng.wikimedia to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/565309 [15:59:00] (03PS2) 10Mforns: Add ng.wikimedia to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/565309 [15:59:49] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Self-merging for deployment train." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/565309 (owner: 10Mforns) [16:19:12] (03PS1) 10Mforns: Update changelog.md for v0.0.112 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/565314 [16:20:17] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Self-merging for deployment train." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/565314 (owner: 10Mforns) [16:28:17] 10Analytics, 10serviceops: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Neil_P._Quinn_WMF) [16:33:38] 10Analytics, 10Analytics-Kanban, 10serviceops: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) [17:00:25] !log deployed refinery-source v0.0.112 [17:00:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:00:47] standup? [17:01:04] 10Analytics, 10Better Use Of Data, 10Epic, 10Patch-For-Review, and 2 others: Prototype client to log errors in vagrant - https://phabricator.wikimedia.org/T235189 (10phuedx) > @phuedx should hang out with us and opine on the interface and what devs would like to see from this client I was OoO November thr... [17:02:32] 10Analytics, 10Better Use Of Data, 10Epic, 10Patch-For-Review, and 2 others: Prototype client to log errors in vagrant - https://phabricator.wikimedia.org/T235189 (10Nuria) @phuedux please do chime in on CR, that would be best as we are aiming to merge this code next week. [17:03:26] 10Analytics, 10Better Use Of Data, 10Epic, 10Patch-For-Review, and 2 others: Prototype client to log errors in vagrant - https://phabricator.wikimedia.org/T235189 (10Nuria) Added you to CR: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/553376/ [17:04:23] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10User-Elukey: Upgrade to Superset 0.35.2 - https://phabricator.wikimedia.org/T242870 (10elukey) [17:06:21] 10Analytics: Data quality Dashboards 2.0 - https://phabricator.wikimedia.org/T242995 (10Nuria) [17:21:56] 10Analytics, 10Analytics-Kanban, 10serviceops: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10akosiaris) >>! In T242861#5809439, @Ottomata wrote: >> Why mess with wmf.releasename > > Because wmf.releasename doesn't currently c... [17:26:23] 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10User-Elukey: Upgrade to Superset 0.35.2 - https://phabricator.wikimedia.org/T242870 (10cchen) Thank you @Nuria! I just tried the new feature on the dashboard, very useful! [17:40:53] 10Analytics: VPN access to superset/turnilo instead of LDAP - https://phabricator.wikimedia.org/T242998 (10Nuria) [17:40:58] 10Analytics: Requesting kerberos access for jwang - https://phabricator.wikimedia.org/T242813 (10jwang) Hi Expect, I want to update my shell name, which was changed in other ticket (https://phabricator.wikimedia.org/T242807) shell name is jiawang Thanks, Jennifer [17:45:16] 10Analytics: VPN access to superset/turnilo instead of LDAP - https://phabricator.wikimedia.org/T242998 (10Nuria) VPN access to superset/turnilo instead of LDAP user/password authentication Just a cc to #security to know whether we have any plans to have a VPN for employees at WMF and WMDE [18:00:11] ping joal: you joining for groskin? We are doing retro [18:22:44] 10Analytics: [EventLoggingToDruid] Add support for ingesting subfields of map columns - https://phabricator.wikimedia.org/T208589 (10Ottomata) a:05mfornsβ†’03Ottomata I'm going to find some time to work on this. [18:22:59] 10Analytics, 10Analytics-Kanban: [EventLoggingToDruid] Add support for ingesting subfields of map columns - https://phabricator.wikimedia.org/T208589 (10Ottomata) [18:39:06] 10Analytics, 10Analytics-Kanban, 10serviceops: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) > My question is more on the line of why use wmf.releasename to identify the service to begin with. We can just use service... [18:41:11] Hi team - joining back [18:41:17] sorry for the missed ping nuria :( [18:49:20] (03CR) 10Milimetric: "such strong opinions, I like it." (036 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) (owner: 10Fdans) [18:49:49] nuria: can we talk now, or later? [18:51:13] (03CR) 10Milimetric: "we should remove unused strings, either here or in a follow-up change, otherwise +1 on the code, I'll wait for your next patch to test UI" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/558702 (https://phabricator.wikimedia.org/T240617) (owner: 10Fdans) [18:59:44] * elukey off! [19:05:18] milimetric: hi :) WOuld you have a minute to talk about flink/rest API? [19:35:17] nuria: gist updated with numbers from all referenced types (code updated as well) [19:50:07] joal: sorry I just saw your ping, sure but it’s late? [19:50:18] it's ok :) [19:50:38] milimetric: if you have some time now, let's spend a minute :) [20:11:43] (03CR) 10Mforns: [V: 03+2] Add anomaly detection to data quality stats workflow [analytics/refinery] - 10https://gerrit.wikimedia.org/r/563200 (https://phabricator.wikimedia.org/T235486) (owner: 10Mforns) [20:17:59] 10Analytics, 10Analytics-Kanban, 10serviceops: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) Too bad set based selectors [[ https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#service-and-replic... [20:52:26] !log deployed refinery accompanying source v0.0.112 [20:52:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:53:01] jo: running this now (couldn't sql and talk to you, you require too much brain % :)) [20:53:05] https://www.irccloud.com/pastebin/6qVhFK2E/ [20:53:12] 913724 [20:53:15] you were right!!!! [20:54:08] it's so close to a million that now I'm worried about the data content in your bloodstream to give you such precise intuition :) [20:55:07] :D [20:55:09] so yeah, we could easily use this as a filter, load data for these pages in one datasource and data for the rest in a separate datasource [20:56:05] dropping for tonight :) see you tomorrow team [21:11:54] bye joal! [21:55:40] 10Analytics, 10Analytics-Kanban, 10serviceops: Clarify multi-service instance concepts in helm charts and enable canary releases - https://phabricator.wikimedia.org/T242861 (10Ottomata) I just updated [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/564052 | my patch ]]; I'll explain my new...