[06:30:08] Hi team [06:30:19] bonjour [07:13:47] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10ayounsi) In eqiad it is 208.80.154.32 / 2620:0:861:1:208:80:154:32. But this is subject to changes so it would be better to not hardcode IPs. Instead it's better to use ranges, like 208.80.154.0/2... [07:25:58] 10Analytics-Clusters: Create the new Hadoop test cluster - https://phabricator.wikimedia.org/T255139 (10elukey) The new hosts are ready to be used: - an-test-master100[1,2] will be the Hadoop master nodes - an-test-coord1001 will be the coordinator/launcher node - an-test-worker100[1-3] will be the Hadoop worke... [07:28:03] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10RhinosF1) I would imagine it's an entry in [https://github.com/wikimedia/operations-mediawiki-config/blob/e0969147a87cdb961c07abd39d7dd66554e9b58c/wmf-config/reverse-proxy.php#L5](this list) becau... [07:29:49] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10elukey) https://meta.wikimedia.org/wiki/Wikimedia_URL_Shortener#Security > Users or IPs who cannot edit Meta-Wiki (blocked) are not allowed to create short URLs [07:32:59] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10RhinosF1) >>! In T233336#6414979, @elukey wrote: > https://meta.wikimedia.org/wiki/Wikimedia_URL_Shortener#Security > >> Users or IPs who cannot edit Meta-Wiki (blocked) are not allowed to create... [09:06:17] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [BUG] Presto Error: parquet file declares column as wrong type for wmf..mediawiki_history - https://phabricator.wikimedia.org/T261261 (10elukey) >>! In T261261#6412958, @Nuria wrote: > @elukey to update the setting in presto that assumes order on columns o... [09:40:57] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade schema[12]00[12] to Debian Buster - https://phabricator.wikimedia.org/T255026 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: `schema1001.eqiad.wmnet` - schema1001.eqiad.wmnet (**PASS**) -... [09:43:01] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade schema[12]00[12] to Debian Buster - https://phabricator.wikimedia.org/T255026 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: `schema1002.eqiad.wmnet` - schema1002.eqiad.wmnet (**PASS**) -... [09:44:15] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade schema[12]00[12] to Debian Buster - https://phabricator.wikimedia.org/T255026 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: `schema2001.codfw.wmnet` - schema2001.codfw.wmnet (**PASS**) -... [09:45:09] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade schema[12]00[12] to Debian Buster - https://phabricator.wikimedia.org/T255026 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: `schema2002.codfw.wmnet` - schema2002.codfw.wmnet (**PASS**) -... [09:50:00] 10Analytics, 10Analytics-Kanban: Move the Analytics infrastructure to Debian Buster - https://phabricator.wikimedia.org/T234629 (10elukey) [10:42:04] * elukey lunch [11:17:47] 10Analytics: kerberos identity request - https://phabricator.wikimedia.org/T261393 (10Cparle) [12:30:18] 10Analytics: kerberos identity request - https://phabricator.wikimedia.org/T261393 (10elukey) 05Open→03Invalid Hey @Cparle I am going to fold this task into the main one for access request, hope it is ok! [13:04:47] joal: whenever you have a moment can we bc for a second about netflow? [13:05:00] Hi fdans - now? [13:05:01] (hello) [13:05:06] joal: sure! [13:05:10] omw [13:05:11] OMW :) [13:23:00] 10Analytics: kerberos identity request - https://phabricator.wikimedia.org/T261393 (10Cparle) that's cool, thanks @elukey [13:26:06] 10Analytics-Clusters: AMD ROCm kernel drivers on stat1005/stat1008 don't support some features - https://phabricator.wikimedia.org/T260442 (10MoritzMuehlenhoff) There is 5.7.10-1~bpo10+1 in buster-backports, but it comes with many downsides and I don't think any of those warrant the sysfs addition; there can be... [13:29:34] !log restart jvm daemons on analytics1042, aqs1004, kafka-jumbo1001 to pick up new openjdk upgrades (canaries) [13:29:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:36:20] 10Analytics-Clusters: Create the new Hadoop test cluster - https://phabricator.wikimedia.org/T255139 (10Cmjohnson) [13:37:13] 10Analytics-Clusters: Create the new Hadoop test cluster - https://phabricator.wikimedia.org/T255139 (10Cmjohnson) [13:43:27] joal: one thought: what if the augmentation only happened at the batch/refine step, but not at the real time step? could this be an option to bring up to the task too? [13:43:57] ie we add these dimensions but they're null when they first come through the real time jobs, but they are populated later at the second pass [13:46:05] it's a possibility fdans indeed [13:50:50] 10Analytics-Clusters, 10Analytics-Radar, 10Operations, 10observability, 10Patch-For-Review: Move kafkamon hosts to Debian Buster - https://phabricator.wikimedia.org/T252773 (10elukey) @herron Hi! What is the status of the task? Anything that I can help with? [13:58:30] taking a break before standup [14:20:40] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10fdans) @JAllemandou and I just had a chat about these changes. Before proceeding with any of the ways Joseph described above, @faidon: how importa... [14:22:59] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops: Add more dimensions in the netflow/pmacct/Druid pipeline - https://phabricator.wikimedia.org/T254332 (10CDanis) It's critical that this data remain real-time, even if some of the fields aren't available in the real-time data. [14:44:33] 10Analytics, 10Event-Platform, 10Patch-For-Review: Allow analytics VLAN to reach all eventgate instances in all DCs - https://phabricator.wikimedia.org/T261356 (10elukey) Just deployed the changes, lemme know if everything works as expected! [14:50:13] new openjdk deployed, that means roll restart of all daemons :) [14:51:20] razzi: good morning :) we can talk about --^ when you have some time, I'll show you all the joy of this task :) [14:57:01] a-team: sorry, kid’s doctor appointment ran late, will miss standup. Yesterday I had tech com, fixed up the proxy for the url shortner but still ran into some config problem (I think it’s some shortcoming of the proxy itself maybe?), and worked on removing and renaming “breakdown” to “dimension” around wikistats. I should probably explain more about why this is needed, but I’ll continue working on that. [15:08:20] 10Analytics-Clusters, 10Analytics-Radar, 10Operations, 10observability, 10Patch-For-Review: Move kafkamon hosts to Debian Buster - https://phabricator.wikimedia.org/T252773 (10herron) [15:10:39] 10Analytics-Clusters, 10Analytics-Radar, 10Operations, 10observability, 10Patch-For-Review: Move kafkamon hosts to Debian Buster - https://phabricator.wikimedia.org/T252773 (10herron) Hey @elukey, prep work is done for the new hosts. Will be performing cut-over in the near future, will keep you on the cc. [15:31:27] ottomata: I will have two pretty trivial patches to eventgate for you shortly, but, here's a Network Error report against a personal domain that was processed by a dev eventgate: https://phabricator.wikimedia.org/P12405 [15:34:09] hmmmmm [15:34:21] too bad we can't get that user_agent into a different field hmmmm [15:34:25] hmmmm [15:46:19] 10Analytics-Clusters, 10Analytics-Radar, 10Operations, 10observability, 10Patch-For-Review: Move kafkamon hosts to Debian Buster - https://phabricator.wikimedia.org/T252773 (10herron) [15:48:51] https://github.com/wikimedia/eventgate/pull/10 :) [15:49:13] 10Analytics, 10Analytics-Kanban, 10Event-Platform: Allow analytics VLAN to reach all eventgate instances in all DCs - https://phabricator.wikimedia.org/T261356 (10Milimetric) 05Open→03Resolved a:03Milimetric [15:49:16] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events - https://phabricator.wikimedia.org/T251609 (10Milimetric) [15:51:21] 10Analytics: Update data-purge for processed mediawiki_wikitext_history (6 snapshot kept, 3 would be sufficient) - https://phabricator.wikimedia.org/T237047 (10Milimetric) p:05Medium→03Unbreak! [15:51:39] 10Analytics: Update data-purge for processed mediawiki_wikitext_history (6 snapshot kept, 3 would be sufficient) - https://phabricator.wikimedia.org/T237047 (10Milimetric) p:05Unbreak!→03Triage [15:51:49] 10Analytics, 10Analytics-Kanban: Update data-purge for processed mediawiki_wikitext_history (6 snapshot kept, 3 would be sufficient) - https://phabricator.wikimedia.org/T237047 (10Milimetric) p:05Triage→03High a:03JAllemandou [15:59:07] 10Analytics: Review current usage of HDFS and establish what/if data can be dropped periodically - https://phabricator.wikimedia.org/T261283 (10Milimetric) p:05Triage→03Medium [16:03:34] 10Analytics: Review current usage of HDFS and establish what/if data can be dropped periodically - https://phabricator.wikimedia.org/T261283 (10Milimetric) idea: make the data available through superset in order to plot a tree-map ala https://bl.ocks.org/ganezasan/52fced34d2182483995f0ca3960fe228 [16:04:18] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [BUG] Presto Error: parquet file declares column as wrong type for wmf..mediawiki_history - https://phabricator.wikimedia.org/T261261 (10Milimetric) p:05Triage→03High [16:04:40] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [BUG] Presto Error: parquet file declares column as wrong type for wmf..mediawiki_history - https://phabricator.wikimedia.org/T261261 (10Milimetric) 05Open→03Resolved @SNowick_WMF please reopen if you see something wrong [16:14:52] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Product-Infrastructure-Data: Streams with empty configs should be rendered as {} in the JSON returned by StreamConfig API - https://phabricator.wikimedia.org/T259917 (10mpopov) p:05Low→03Unbreak! Okay, it seems `destination_event_service` is now omit... [16:20:28] 10Analytics, 10Analytics-Wikistats: Add "Top linked article" metric - https://phabricator.wikimedia.org/T220484 (10Milimetric) 05Open→03Declined We're looking for solid use cases and stakeholders, so we're cleaning up old tasks that don't have that. Do re-open if you feel you can own the task and need the... [16:21:02] 10Analytics, 10Analytics-Wikistats: Add "Number of stub articles" metric - https://phabricator.wikimedia.org/T220483 (10Milimetric) We're looking for solid use cases and stakeholders, so we're cleaning up old tasks that don't have that. Do re-open if you feel you can own the task and need the outcome. [16:21:04] 10Analytics, 10Analytics-Wikistats: Add "Number of stub articles" metric - https://phabricator.wikimedia.org/T220483 (10Milimetric) 05Open→03Declined [16:21:06] 10Analytics: Epic: content-based metrics - https://phabricator.wikimedia.org/T220717 (10Milimetric) [16:21:15] 10Analytics, 10Analytics-Wikistats: Add "Top large articles" metric - https://phabricator.wikimedia.org/T220482 (10Milimetric) 05Open→03Declined We're looking for solid use cases and stakeholders, so we're cleaning up old tasks that don't have that. Do re-open if you feel you can own the task and need the... [16:21:20] 10Analytics: Epic: content-based metrics - https://phabricator.wikimedia.org/T220717 (10Milimetric) [16:25:45] 10Analytics: Update datasets to have explicit timestamp for druid indexation facilitation - https://phabricator.wikimedia.org/T205617 (10Milimetric) p:05Triage→03Medium [16:29:46] 10Analytics: Hive log4j logging is misconfigured - https://phabricator.wikimedia.org/T216294 (10Milimetric) p:05Triage→03Low @elukey's going to try and fix this if it's not too full of sharks [16:30:08] 10Analytics: Survey dashboard layout for dashiki - https://phabricator.wikimedia.org/T178399 (10Milimetric) 05Open→03Declined We're looking for solid use cases and stakeholders, so we're cleaning up old tasks that don't have that. Do re-open if you feel you can own the task and need the outcome. [16:31:58] elukey: TIL - fsimage_0000000004675023290 [16:32:01] oops [16:32:08] elukey: https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html [16:32:19] This was what I was after [16:34:23] ahhh nice! [16:34:28] TIL as well! [16:34:44] elukey: I'm gonna investigate this and see [16:51:22] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: [BUG] Presto Error: parquet file declares column as wrong type for wmf..mediawiki_history - https://phabricator.wikimedia.org/T261261 (10SNowick_WMF) Looks good, queries are working now, thanks! [16:52:13] going offline, ttl! [17:37:21] 10Analytics, 10Analytics-Kanban, 10Platform Team Workboards (Initiatives): reportupdater Pingback reports are broken and need to be refactored - https://phabricator.wikimedia.org/T246154 (10mforns) Now they have finished updating, yes they look better now! I will now re-run the reports from 2018-09-02 (not 2... [18:02:32] (03PS1) 10Mforns: Update changelog.md for v0.0.134 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622860 [18:03:03] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Self-merging for deployment train." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622860 (owner: 10Mforns) [18:03:51] Starting build #57 for job analytics-refinery-maven-release-docker [18:15:07] Project analytics-refinery-maven-release-docker build #57: 09SUCCESS in 11 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/57/ [18:28:43] Starting build #24 for job analytics-refinery-update-jars-docker [18:28:59] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.0.134 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/622862 [18:29:00] Project analytics-refinery-update-jars-docker build #24: 09SUCCESS in 17 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/24/ [18:30:16] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Merging for deployment train" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/622862 (owner: 10Maven-release-user) [18:30:36] !log deployed refinery-source v0.0.134 [18:30:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:32:24] (03PS2) 10Mforns: Update mediawiki-history-dumps parameters [analytics/refinery] - 10https://gerrit.wikimedia.org/r/621702 (https://phabricator.wikimedia.org/T254233) (owner: 10Joal) [18:33:06] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Merging for deployment train" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/621702 (https://phabricator.wikimedia.org/T254233) (owner: 10Joal) [18:35:36] nuria: yt? if so got a sec to brain bounce some canary event things? [18:35:54] ottomata: on meeeting but shall be done in a bit [18:40:38] k [18:41:45] !log starting refinery deploy (ref v0.0.134) [18:41:47] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:44:49] ottomata: bc? [18:46:55] ya [18:46:57] coming [19:01:32] fdans: thought of one more thing on the jsonschema-tools task [19:01:32] https://github.com/wikimedia/jsonschema-tools/pull/16#pullrequestreview-476989912 [19:05:29] !log finished refinery deploy (ref v0.0.134) [19:05:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:45:11] ottomata: thank you for merging! [20:45:23] tis deployed as well, sorry for the delay! [20:45:29] i see your new streams :) [20:46:24] thanks!!! [20:47:52] (03PS7) 10Milimetric: [WIP] Add filter/split component to Wikistats TODO: I'm pushing this even though I haven't tested it in the UI at all, because it passes tests and I find that interesting, I want to see the difference between this and the eventually working version, to see if I miss adding any tests as I make it work. Basically, this just tries to remove references to "breakdown" from everywhere. [analytics/wikistats2] [20:47:53] (https://phabricator.wikimedia.org/T249758) (owner: 10Fdans) [21:25:45] 10Analytics-Radar, 10Anti-Harassment, 10Product-Analytics: Capture special mute events in Prefupdate tabe - https://phabricator.wikimedia.org/T261461 (10jwang) [21:26:17] 10Analytics-Radar, 10Anti-Harassment, 10Product-Analytics: Capture special mute events in Prefupdate tabe - https://phabricator.wikimedia.org/T261461 (10jwang) [21:31:41] 10Analytics-Radar, 10Anti-Harassment, 10Product-Analytics: Capture special mute events in Prefupdate tabe - https://phabricator.wikimedia.org/T261461 (10jwang) [22:36:19] 10Analytics-Radar, 10Anti-Harassment, 10Product-Analytics: Capture special mute events in Prefupdate tabe - https://phabricator.wikimedia.org/T261461 (10jwang) [22:37:26] 10Analytics-Radar, 10Anti-Harassment, 10Product-Analytics: Capture special mute events in Prefupdate tabe - https://phabricator.wikimedia.org/T261461 (10jwang)