[00:07:19] 10Analytics, 10Analytics-Kanban: POC More efficient Bot filtering on pageview data - https://phabricator.wikimedia.org/T211359 (10leila) [00:11:31] 10Analytics, 10DBA, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10Tbayer) The [[https://www.mediawiki.org/wiki/Manual:Page_props_table |page_props table]] contains `wikibase_item` values for a give... [00:16:55] 10Analytics, 10DBA, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10diego) @Tbayer , great. Thanks. [00:33:48] 10Analytics: Alarms in Eventlogging hadoop sanitization - https://phabricator.wikimedia.org/T198910 (10Nuria) Alarms via refinemonitor I think cover hadoop sanitization as if raw data is present in the eventlogging raw directories refine monitor will alarm if those have not been refined. Now, do we equally alarm... [00:35:43] 10Analytics, 10Analytics-EventLogging: Alarm on errors on /var/log/upstart/eventlogging* files - https://phabricator.wikimedia.org/T170620 (10Nuria) I think alarms for process not restarting have been taken care off with the systemd migration can @elukey confirm and close ticket if pertains? [03:24:35] 10Analytics: Alarms in Eventlogging hadoop sanitization - https://phabricator.wikimedia.org/T198910 (10Ottomata) Ya: https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-camus/src/main/scala/org/wikimedia/analytics/refinery/camus/CamusPartitionChecker.scala#L120-L122 https://github.com/w... [03:52:22] 10Analytics: Data Quality Alarms - https://phabricator.wikimedia.org/T198986 (10Nuria) [03:52:25] 10Analytics: Alarms in Eventlogging hadoop sanitization - https://phabricator.wikimedia.org/T198910 (10Nuria) 05Open→03Resolved [04:06:21] 10Analytics, 10Cloud-Services: Rethink Cloud DB replicas - https://phabricator.wikimedia.org/T215858 (10Milimetric) [05:06:04] 10Analytics: Coarse alarm on data quality for refined data based on entrophy calculations - https://phabricator.wikimedia.org/T215863 (10Nuria) p:05Triage→03Normal [07:18:39] morning! [07:23:04] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10chelsyx) @santhosh Thanks so much for the event logging! https://meta.wikimedia.org/w/index.php?title=Schema:ExternalGuidance&ol... [07:28:52] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) @ayounsi modified a bit analytics-in4 term mysql-dbstore: ` [edit firewall family inet filter analytics-in4 t... [07:53:39] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10santhosh) >>! In T212414#4946442, @chelsyx wrote: > @santhosh Thanks so much for the event logging! https://meta.wikimedia.org/w... [08:07:22] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10chelsyx) >>! In T212414#4946460, @santhosh wrote: >>>! In T212414#4946442, @chelsyx wrote: >> @santhosh Thanks so much for the e... [08:34:06] https://www.amd.com/en/products/professional-graphics/instinct-mi60 looks nice :D [08:37:52] this has been released in November [08:37:56] so should be the last one [08:38:02] and fully supported by rocm [08:39:41] but there is a caveat - those cards require PCI 3.0 and support for PCIe atomics [08:46:38] but for our card, Hawaii, is not required [09:02:17] 10Analytics, 10Analytics-Kanban, 10User-Marostegui: Migrate users to dbstore100[3-5] - https://phabricator.wikimedia.org/T215589 (10elukey) Message to everybody: Analytics and the Data Persistence team are planning to schedule the official cut off date for the `staging` database on dbstore1002 for Monday 18... [09:03:02] 10Analytics, 10Operations, 10WMF-Legal, 10Privacy: Honor DNT header for access logs & varnish logs - https://phabricator.wikimedia.org/T98831 (10Gilles) 05Open→03Declined Sure, there will probably be more precise standards in the future for this sort of thing. [09:16:58] 10Analytics, 10DBA, 10MediaWiki-Database, 10Research, 10Wikidata: Improve interlingual links across wikis through Wikidata IDs - https://phabricator.wikimedia.org/T215616 (10jcrespo) > the API works good for query specific pages/entities, not for example to know which pages that existing in X_wiki are mi... [09:17:08] (03PS1) 10Fdans: Update dependencies to solve vulnerabilities [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/489998 (https://phabricator.wikimedia.org/T210589) [09:35:01] (03PS4) 10Fdans: Change email send workflow to notify of completed jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484657 (https://phabricator.wikimedia.org/T206894) [09:39:35] !log systemctl disable/stop mediawiki-geoeditors-drop-month.timer on an-coord1001 [09:39:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:40:54] 10Analytics, 10monitoring: Grafana shows zero EventLogging events for around 44 hours around January 15 - https://phabricator.wikimedia.org/T215744 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi It is indeed, thanks @Volans ! Tentatively resolving [09:42:46] 10Analytics, 10Analytics-Kanban, 10User-Marostegui: Migrate users to dbstore100[3-5] - https://phabricator.wikimedia.org/T215589 (10Marostegui) Read only time would be around 16h (T210478#4942371) [09:53:01] (03PS5) 10Fdans: Change email send workflow to notify of completed jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484657 (https://phabricator.wikimedia.org/T206894) [09:56:18] (03CR) 10Fdans: "Tested successfully with mediawiki history denormalized" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484657 (https://phabricator.wikimedia.org/T206894) (owner: 10Fdans) [11:16:51] 10Analytics: Explain with annotations start of new registered users data - https://phabricator.wikimedia.org/T215887 (10fdans) [11:27:20] 10Analytics: Project families are not available in pages to date metric in Wikistats UI - https://phabricator.wikimedia.org/T215889 (10fdans) [11:27:29] 10Analytics, 10Analytics-Kanban: Project families are not available in pages to date metric in Wikistats UI - https://phabricator.wikimedia.org/T215889 (10fdans) [11:27:45] 10Analytics, 10Analytics-Kanban: Project families are not available in pages to date metric in Wikistats UI - https://phabricator.wikimedia.org/T215889 (10fdans) p:05Triage→03High [11:29:13] (03PS1) 10Fdans: Add globalFamily property to pages to date config [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/490029 (https://phabricator.wikimedia.org/T189626) [11:40:13] elukey@stat1005:~$ sudo dmesg | grep amdgpu [11:40:14] [ 17.830797] [drm] amdgpu kernel modesetting enabled. [11:40:18] this is encouraging :) [11:46:11] 10Analytics, 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Allow Erik Bernhardson to have root access on stat1005 for GPU testing - https://phabricator.wikimedia.org/T215384 (10MoritzMuehlenhoff) 05Open→03Resolved stat1005 is now running Debian buster and I've enabled Erik's access. [11:47:10] 10Analytics, 10Operations, 10Research-management, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10MoritzMuehlenhoff) > TODOS: > * stat1005 will be reimaged to Debian Stretch when the SRE team is ready (work is currently in progress to import Buster in production)... [11:51:57] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Check wikistats numbers for agreggations for "all-wikipedias" - https://phabricator.wikimedia.org/T189626 (10fdans) Here is my vetting report so far: https://wikitech.wikimedia.org/wiki/User:Fdans/Vetting_data_lake_metrics_for_project_families [11:53:48] 10Analytics, 10Operations, 10Research-management, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) ` elukey@stat1005:~$ sudo dmesg | grep amdgpu [ 17.830797] [drm] amdgpu kernel modesetting enabled. ` \o/ Following https://rocm.github.io/ROCmInstall.ht... [11:53:50] (03PS2) 10Fdans: Add globalFamily property to pages to date config [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/490029 (https://phabricator.wikimedia.org/T215889) [12:39:47] 10Analytics, 10Operations, 10Research-management, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10MoritzMuehlenhoff) There are no source packages for the debs, given that they seem are otherwise pretty focused on FLOSS (e.g. https://rocm.github.io/ROCmInstall.ht... [12:49:37] 10Analytics, 10Operations, 10Research-management, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10MoritzMuehlenhoff) >>! In T148843#4947165, @MoritzMuehlenhoff wrote: > we should ask them to also publish them. Did that in https://github.com/RadeonOpenCompute/ROC... [12:52:31] * elukey lunch!/ [13:19:40] Heya team - I'm sorry I can't work today, I blocked my back this morning (scapula-level - first time it ever happens to me ...) amd I'm desesperately trying to get a doctor appointment :( [13:39:19] 10Analytics-Kanban, 10Product-Analytics: Make aggregate data on editors per country per wiki publicly available - https://phabricator.wikimedia.org/T131280 (10Neil_P._Quinn_WMF) [14:21:16] joal: :( [14:27:45] hey teammmm :] [14:34:25] hola! [14:35:13] o/ [14:38:28] (03CR) 10Mforns: [C: 03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484657 (https://phabricator.wikimedia.org/T206894) (owner: 10Fdans) [14:45:18] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Halfak) Is this specific to ORES or is it general to anyone who is loading this type of data into hadoop via event bus? Ping @Ottomata.... [14:54:19] hell yea profile photo club ottomata ! [14:54:37] haha i need a different one! this one is from 2011 [14:54:44] i just switched to Franz (carolyn told me about it) [14:54:45] trying it out [14:54:49] but that means i'm using IRCcloud for the first time [14:54:56] which is....pretty nice after all? [14:55:22] I'm not yet used to reading it so I'm worried i'll miss some notifications but I'll get used to it! [14:56:00] (03CR) 10Mforns: [C: 04-1] "I get errors when trying to resintall with:" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/489998 (https://phabricator.wikimedia.org/T210589) (owner: 10Fdans) [14:56:53] I notice what seems like generic bouncer functionality in irccloud beta functionality [14:56:55] but haven't tried it [14:58:21] ottomata: I used Franz for a while, it looooves to use all of the battery [15:04:39] ottomata: o/ [15:04:55] I am a bit confused by the sample => 10 and then the /1000 division [15:05:06] mforns: why are you removing the semantic dir? [15:06:30] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Ottomata) Ya not specific to ORES. This would affect anyone trying to schedule Oozie jobs on data imported periodically from Kafka. @P... [15:07:51] fdans, to test installation from scratch [15:08:46] mforns: but the semantic dir comes with the repo [15:08:55] fdans, in any case, I get the same error [15:08:56] not that I think that's the issue here though [15:09:05] just by deleting node_modules [15:09:29] not sure it needs to come with the repo no? [15:09:41] it should be installed with npm install [15:10:10] maybe I miss sth, but it still fails for me when just deleting node_modules [15:21:48] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/490029 (https://phabricator.wikimedia.org/T215889) (owner: 10Fdans) [15:24:13] (03Merged) 10jenkins-bot: Add globalFamily property to pages to date config [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/490029 (https://phabricator.wikimedia.org/T215889) (owner: 10Fdans) [15:27:05] 10Analytics, 10Operations, 10Research-management, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) Let's see if we can narrow down the packages needed: ` \-- rocm-dev |--hsa-rocr-dev |--hsa-ext-rocr-dev |--rocm-device-libs |--rocm... [15:28:30] mforns: ah it was a package lock issue, deleting it solves it [15:28:33] updating patch [15:28:40] cool! [15:28:53] (03PS2) 10Fdans: Update dependencies to solve vulnerabilities [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/489998 (https://phabricator.wikimedia.org/T210589) [15:32:22] fdans, hm, I get the same error O.o [15:32:52] mforns: checking out or removing package lock? [15:36:55] checking out [15:36:58] fdans, ^ [15:38:58] mforns: and it persists if you delete package lock? [15:39:27] trying [15:40:21] fdans, yes, persists [15:41:01] ‎(ﾉಥ益ಥ）ﾉ ┻━┻ [15:41:40] xD [15:46:21] 10Analytics, 10Operations, 10Research-management, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) Nice summary in https://rocm-documentation.readthedocs.io/en/latest/Programming_Guides/Programming-Guides.html [15:48:04] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Pchelolo) > ORES change propagation is configured to mostly go through one datacenter Which of the ORES-related CP rules are you referrin... [15:48:45] (03PS3) 10Fdans: Update dependencies to solve vulnerabilities [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/489998 (https://phabricator.wikimedia.org/T210589) [15:49:03] mforns: if you try it rm semantic now pls [15:49:11] sure [15:50:18] fdans, it vvvvvorks! [15:50:39] ugh, I hated every part of that [15:50:43] I'm glad it works [15:52:48] :D [15:53:24] now I'm trying to execute tests, but don't remember how... in the README says it should be karma start, but.. that returns an error [15:54:33] wait [15:56:03] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current): Wire ORES recent_score events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Pchelolo) There's one more issue that's not being addressed here. The precaching config does not have all the models available for the wiki, s... [15:58:27] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Services (watching): Wire ORES recent_score events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Pchelolo) [16:00:16] ping ottomata , joal, fdans , mforns standdduppp [16:01:35] OH [16:01:36] sorry [16:01:36] ! [16:01:37] coming [16:11:50] 10Analytics, 10Operations, 10Research-management, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) First joy: ` (tensorflow_test) elukey@stat1005:~/tensorflow_test$ pip3 install tensorflow-rocm Collecting tensorflow-rocm Could not find a version that sa... [16:15:03] (03CR) 10Milimetric: Introduce analytics-mysql (035 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/488473 (https://phabricator.wikimedia.org/T212386) (owner: 10Elukey) [16:20:14] 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10Miriam) Hi @RyanSteinberg ! We have decided to turn on the data collection for a short period of time, so that you have real data s... [16:38:00] (03PS1) 10Ladsgroup: Add methods for new hosts and changing good_articles.php to use that [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/490088 (https://phabricator.wikimedia.org/T213894) [16:42:07] 10Analytics, 10Operations, 10Research-management, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10Miriam) [16:42:11] 10Analytics, 10Discovery-Search, 10Multimedia, 10Reading-Admin, and 3 others: Image Classification Working Group - https://phabricator.wikimedia.org/T215413 (10Miriam) [16:42:30] 10Analytics, 10Discovery-Search, 10Multimedia, 10Reading-Admin, and 3 others: Image Classification Working Group - https://phabricator.wikimedia.org/T215413 (10Miriam) [17:09:59] (03CR) 10Ottomata: [C: 03+1] Lowercase capsule fields in EL sanitization whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/489817 (https://phabricator.wikimedia.org/T209503) (owner: 10Mforns) [17:35:53] (03PS1) 10Ladsgroup: Add methods for new hosts and changing good_articles.php to use that [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/490105 (https://phabricator.wikimedia.org/T213894) [17:38:11] (03CR) 10Mforns: Make saltrotate store salts with timestamps as file name. (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484250 (https://phabricator.wikimedia.org/T212014) (owner: 10Mforns) [17:38:24] (03CR) 10Ottomata: [C: 03+1] "Discussed with Marcel some ideas about using local file system and pushing to HDFS, but +1 for this as is!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484250 (https://phabricator.wikimedia.org/T212014) (owner: 10Mforns) [17:40:10] (03CR) 10Mforns: "Also add some unit tests below" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484250 (https://phabricator.wikimedia.org/T212014) (owner: 10Mforns) [17:54:56] (03CR) 10Ottomata: [C: 03+1] "Only comment: maybe move the getParittionStart stuff into HivePartition, if you like!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/484708 (https://phabricator.wikimedia.org/T212014) (owner: 10Mforns) [18:17:09] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Ottomata) It isn't different from any other event. The issue is that Adam is building an Oozie job that gets launched based on full exist... [18:20:33] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Pchelolo) So in this case we have all the events in eqiad, and some rogue events in codfw. Is it possible to consume from 2 topics in the... [18:27:57] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Ottomata) Ah, no sorry, let me try to be clearer. The problem here is that there are NO events in codfw, but the job needs to handle the... [18:28:05] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 3 others: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10Neil_P._Quinn_WMF) >>! In T212386#4943549, @elukey wrote: > @Neil_P._Quinn_WMF... [18:28:50] 10Analytics, 10User-Elukey: Restoring the daily traffic anomaly reports - https://phabricator.wikimedia.org/T215379 (10Tbayer) >>! In T215379#4940339, @elukey wrote: > No problem! Thanks for clarifying :) > >>>! In T215379#4940138, @Tbayer wrote: >> >> Even given that the report was just based on two users' c... [18:31:00] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Pchelolo) > If one of the topics in a stream has no data for a given hour, we can't tell if that is because of an upstream (producer relat... [18:33:06] ottomata: sorry one last q before logging off! Is 10 or 1000 in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/490067/ ? [18:33:44] elukey yes meant 10! [18:33:45] sorry [18:33:47] 1000! [18:33:54] i had typed the exasmple before i calculated [18:33:54] ahhhhh okok! [18:33:58] 1000 not 10 [18:34:09] all right thanks for the hint, will apply :) [18:34:15] yup! (grabbing coffee...) [18:36:52] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Services (watching): Wire ORES recent_score events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Pchelolo) I'm trying to get alternative solutions because I would honestly hate emitting a heartbeat event. [18:46:47] 10Analytics, 10Operations, 10Research-management, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) Just as test, I downloaded several python3.6 packages from snapshot.debian.org and applied them to stat1005. This is the issue that I am facing: ` elukey@st... [18:47:10] * elukey off! [19:03:08] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Ottomata) > < previous hour Maybe, we'd have to find some way to do that in Oozie, but it might be possible. Even if it is, this would be... [19:21:59] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Pchelolo) Hmm.... Just brainstorming here... This doesn't seem solvable even with the artificial event too. If we emit the artificial ev... [19:37:48] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Ottomata) > If we emit the artificial event via the same machinery and we emit normal events (which we should do) it will get delayed the... [19:56:09] 10Analytics, 10Knowledge-Integrity, 10Research, 10Epic, 10Patch-For-Review: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969 (10RyanSteinberg) Hi @Miriam. This sounds good to me. I should have some time to look at the sampled data this Friday. Thank you! [20:37:23] PROBLEM - eventbus grafana alert on icinga2001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. [20:38:47] ...? [20:40:10] ah [20:40:10] mediawiki.page-links-change [20:43:33] 10Analytics, 10EventBus, 10Research, 10Wikidata, and 5 others: Surface link changes as a stream - https://phabricator.wikimedia.org/T214706 (10Ottomata) Just saw some errors in prod Failed processing event: Topic mediawiki.page-links-change not configured ^ should fix. [20:43:48] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Services (watching): Wire ORES recent_score events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Nuria) >I'm trying to get alternative solutions because I would honestly hate emitting a heartbeat event. I gotta say... [20:44:49] 10Analytics, 10ORES, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Services (watching): Wire ORES recent_score events into Hadoop - https://phabricator.wikimedia.org/T209732 (10Ottomata) Generally agree with both of you. [20:45:13] 10Analytics, 10ORES, 10Scoring-platform-team (Current): Emit synthetic mediawiki.revision-score events for both datacenters - https://phabricator.wikimedia.org/T214545 (10Nuria) Let's spend a bit of time trying to see if we can fix this issue in the consumer end which where the problem is, rather than tryin... [20:46:01] RECOVERY - eventbus grafana alert on icinga2001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. [20:47:27] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 3 others: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10Nuria) I think this ticket can be closed. Work on migrating cluster continues t... [20:47:35] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 2 others: Replace the current multisource analytics-store setup - https://phabricator.wikimedia.org/T172410 (10Nuria) [20:47:39] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 3 others: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10Nuria) 05Open→03Resolved [20:53:19] PROBLEM - eventbus grafana alert on icinga2001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. [20:53:34] should be fixed shortly... [20:53:47] false alarm for new (and slightly misconfigure) page-links-change event [20:56:53] RECOVERY - eventbus grafana alert on icinga2001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. [21:51:17] ottomata: if you are arround would you mind merging one superset change? [22:01:40] ottomata: never mind the dbas took care of it [22:14:17] (03CR) 10Nuria: Change email send workflow to notify of completed jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/484657 (https://phabricator.wikimedia.org/T206894) (owner: 10Fdans) [22:23:34] 10Analytics, 10EventBus, 10Growth-Team, 10MediaWiki-Watchlist, and 6 others: Clear watchlist on enwiki only removes 50 items at a time - https://phabricator.wikimedia.org/T207329 (10kostajh) @Pchelolo unfortunately this is still not working. I can see from the profile trace that the first job is enqueued h... [22:45:27] 10Analytics, 10EventBus, 10Growth-Team, 10MediaWiki-Watchlist, and 6 others: Clear watchlist on enwiki only removes 50 items at a time - https://phabricator.wikimedia.org/T207329 (10Pchelolo) Hm... So, in kafka we have: ` {"database": "enwiki", "mediawiki_signature": "97c895ef8e934b335f74b4d8d57d5b7bebd1d... [23:11:19] ottomata: yt? [23:11:39] ottomata: maybe you know why i cannot create a connection from superset to the new cluster? [23:42:56] (03CR) 10Nuria: "Looks good provided we have tested tests run and dashiki actually works, if we merge this let's please test locally and deploy to the brow" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/489998 (https://phabricator.wikimedia.org/T210589) (owner: 10Fdans) [23:49:03] 10Analytics, 10Product-Analytics, 10Readers-Web-Backlog: Popups schema is keeping page/session id longer than 90 days - https://phabricator.wikimedia.org/T207670 (10kzimmerman) @Nuria can we close out this task? Is there any other work that needs to be done on the Product Analytics side? [23:49:41] 10Analytics, 10Product-Analytics, 10Readers-Web-Backlog: Popups schema is keeping page/session id longer than 90 days - https://phabricator.wikimedia.org/T207670 (10Nuria) 05Open→03Resolved [23:57:42] (03PS1) 10Chelsyx: Add updated field names of MobileWikiAppShareAFact to EventLogging whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/490212 (https://phabricator.wikimedia.org/T209087)