[00:54:04] PROBLEM - Check the last execution of monitor_refine_eventlogging_legacy_failure_flags on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_legacy_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [03:02:20] 10Analytics, 10Growth-Team (Current Sprint), 10Product-Analytics (Kanban): Growth: validate that data is purged after 270 days - https://phabricator.wikimedia.org/T249666 (10MMiller_WMF) 05Open→03Resolved Thank you! [03:02:24] 10Analytics, 10Growth-Team, 10Product-Analytics, 10Patch-For-Review: Growth: implement wider data purge window - https://phabricator.wikimedia.org/T237124 (10MMiller_WMF) [06:26:49] 10Analytics, 10Data-Services, 10cloud-services-team (Kanban): Rethink Cloud DB replicas - https://phabricator.wikimedia.org/T215858 (10Marostegui) [06:27:11] 10Analytics, 10Data-Services, 10cloud-services-team (Kanban): Rethink Cloud DB replicas - https://phabricator.wikimedia.org/T215858 (10Marostegui) [07:32:45] 10Analytics-Radar, 10Operations, 10Patch-For-Review: Move Hue to a Buster VM - https://phabricator.wikimedia.org/T258768 (10elukey) In T233073 I am experimenting with building Hue 4.7.1, that is the latest upstream and it should support python3.7. The build procedure seems good (https://gerrit.wikimedia.org/... [11:53:08] helloooo team! [12:29:53] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Discovery-Search (Current work), 10Patch-For-Review: Integrate Wikimedia Event Utilities with discovery-parent-pom - https://phabricator.wikimedia.org/T260375 (10Gehel) 05Open→03Resolved This seems to be all done. @Ottomata: ping me or re-ope... [12:29:57] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events - https://phabricator.wikimedia.org/T251609 (10Gehel) [12:39:11] (03CR) 10Mforns: [V: 03+2] "Adding Joal, now that he's back :D" [analytics/aqs] - 10https://gerrit.wikimedia.org/r/593660 (https://phabricator.wikimedia.org/T238365) (owner: 10Lex Nasser) [12:57:53] Hurray!!!! The internez is back! [12:58:06] Hi team :) I'm finally back [13:28:10] hello joal welcome back :) [13:28:34] Hi fdans :) How are you? [13:29:07] joal: having a very Monday Monday [13:29:11] did you have a nice break [13:29:56] Yes, holidays were great - Can I help ou make today more like another day? [13:35:17] ottomata: helloooo are you on holiday? [13:35:54] hello! [13:35:55] nopers! [13:38:20] ottomata: ohhh cool can I ask you something on the batcave real quick? [13:40:42] for suure! [13:40:45] logging in [13:40:48] actually 2 mins [13:42:30] 10Analytics-Radar, 10Performance-Team, 10MW-1.36-notes (1.36.0-wmf.4; 2020-08-11): Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10Ottomata) I noticed that there was another field (responseStart) that also had this problem, so I went ahead and made a schema edit to add max and... [13:45:14] hi! [13:45:25] hi joal! [13:45:33] Hi milimetric :) [13:46:59] hey joal, welcome back! [13:47:13] thanks milimetric - how are you? [13:47:16] hey milimetric, welcome too! [13:47:23] hi mforns :) [13:47:24] good thx, hope you too [13:47:40] it's like jumping into icy water, fun but initially overwhelming [13:48:57] mforns / fdans: thanks so much for covering for me last week, I'll take your ops weeks as my own [13:49:11] icy water is fun only for tough romanians [13:49:37] :] [13:50:29] yeah, Dan is to tough romanians as circus bears are to wild polar bears [13:51:17] mforns: I thought the fix was deployed for NavTiming and the validation was working, no? [13:51:18] xD, well I only know 1 tough romanian [13:51:31] I don't know if it's deployed.. [13:51:36] but it's failing still [13:54:04] (catching up, currently on "Andrew updated the schema: https://meta.wikimedia.org/w/index.php?title=Schema:NavigationTiming&action=history") [13:57:38] 10Analytics, 10MediaWiki-REST-API, 10Patch-For-Review, 10Platform Team Sprints Board (Sprint 1), and 2 others: System administrator reviews API usage by client - https://phabricator.wikimedia.org/T251812 (10JMeybohm) [14:01:10] ottomata: I caught up and realized you're deploying right now [14:01:19] I updated https://meta.wikimedia.org/wiki/Schema:ResourceTiming as well [14:01:25] https://meta.wikimedia.org/w/index.php?title=Schema%3AResourceTiming&type=revision&diff=20373828&oldid=18358918 [14:16:01] milimetric: do we have problems with that one? [14:16:14] yes ottomata [14:16:17] huh ok [14:16:22] but not as many as NavigationTiming [14:16:53] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Discovery-Search (Current work), 10Patch-For-Review: Integrate Wikimedia Event Utilities with discovery-parent-pom - https://phabricator.wikimedia.org/T260375 (10Ottomata) Thank you so much! [14:20:53] milimetric: there are a bunch of other integer time fields in ResourceTimiing [14:20:57] should we add max and miin to those too? [14:21:15] i thikn yes [14:21:26] ottomata: I think I kept a record of every error in my gmail and I looked for it, that encodedBody thing was the only one that was over max [14:21:38] I mean, I'd add them by default to every integer [14:21:39] but, it would probably be good to add them anyway? [14:21:49] I'll do that, I have it up [14:21:54] oh i already started :) [14:22:00] naw you go [14:22:04] you got it go milimetric :) [14:23:05] https://meta.wikimedia.org/w/index.php?title=Schema%3AResourceTiming&type=revision&diff=20373861&oldid=18358918 [14:25:57] milimetric: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/620711 [14:27:42] ohh no i need to combine [14:29:41] das better [14:29:56] ottomata: you need a comma after that last square bracket, according to convention [14:29:59] otherwise looks good [14:30:21] (I tried editing but conflicted with your change and that file is gigantic) [14:30:25] done [14:30:39] https://integration.wikimedia.org/ci/job/operations-mw-config-php72-composer-diffConfig-docker/3830/console [14:31:41] k, that's merged, when's the swat window? [14:31:48] i will deploy it now [14:32:08] hard to test it though i think because its sampled? [14:32:14] i guess let's just watch validation logs [14:34:09] yeah, it should basically just not create any more validation errors and stop the refine errors [14:34:20] btw have yall seen https://www.kubeflow.org/#overview? [14:34:24] ya [14:34:47] milimetric: [14:34:47] https://phabricator.wikimedia.org/T247564 [14:34:55] also [14:34:56] https://phabricator.wikimedia.org/T226193 [14:35:43] ah very cool [14:38:28] milimetric: i think things look ok [14:38:40] hmm maybe we can check and make sure we are getting events with that schema rev [14:38:52] I'll check kafka [14:40:23] i mean, its only group0 right now [14:40:28] that's things like wiktionarey etc. [14:40:45] ah, that makes sense, I'm getting all kinds of revIds [14:40:57] ok lets go all [14:40:58] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/620713 [14:41:12] looks good https://integration.wikimedia.org/ci/job/operations-mw-config-php72-composer-diffConfig-docker/3832/console [14:42:04] ok milimetric ^ ? [14:42:41] ty [14:42:46] yep, was just double thinking [14:44:31] hm [14:44:39] not changing... [14:45:07] i mean i guess browsers out there need to get updated eventually [14:45:12] buutu i'd expect to see some... [14:49:58] hmm, milimetric i just manually logEvented a NavTiming an event on testwiki [14:50:03] and it used the right ervision [14:50:20] sooo, maybe it just takes a while? [14:55:50] HMM bnut on enwiki it did not? [14:56:26] OH i didn't rebase [14:56:30] i do that so often [14:57:01] trying again [14:58:51] ok they are starting to use new schema rev now [15:00:06] brt getting some more coffee... [15:22:29] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker10[18-40] - https://phabricator.wikimedia.org/T260445 (10Ottomata) [15:22:44] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) upgrade/replace memory in stat100[58] - https://phabricator.wikimedia.org/T260448 (10Ottomata) [15:25:17] 10Analytics, 10CAS-SSO: Allow login to JupyterHub via CAS - https://phabricator.wikimedia.org/T260386 (10Ottomata) p:05Triage→03Medium [15:30:04] 10Analytics, 10Analytics-Kanban, 10Privacy Engineering, 10Product-Analytics, and 3 others: Drop data from Prefupdate schema that is older than 90 days - https://phabricator.wikimedia.org/T250049 (10Ottomata) a:05fdans→03Milimetric [15:34:44] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Data, 10Product-Infrastructure-Team-Backlog: Develop test environment solution for MEP analytics events - https://phabricator.wikimedia.org/T238837 (10Ottomata) [15:34:48] 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 3 others: Set up an instance of EventStreams in beta that will allow for consuming any stream - https://phabricator.wikimedia.org/T253069 (10Ottomata) [15:36:04] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Beta-Cluster-Infrastructure: [beta] EventLogging trying to fetch wrong Schema title - https://phabricator.wikimedia.org/T254058 (10Ottomata) 05Open→03Declined Closing for now, re-open if this is still happening. [15:37:56] 10Analytics: Investigate showing realtime the eventlogging banner stream (currently sampled at 1%) - https://phabricator.wikimedia.org/T255446 (10Ottomata) [15:42:31] 10Analytics, 10Analytics-Kanban, 10Cloud-VPS, 10Puppet: Puppet failing on wikistats.analytics.eqiad.wmflabs: /usr/local/sbin/x509-bundle error - https://phabricator.wikimedia.org/T255464 (10Ottomata) 05Open→03Resolved We realized today that we don't need this instance at all. Deleted it. [15:43:40] 10Analytics, 10Analytics-Kanban, 10MediaWiki-Vagrant: Vagrant's /var/log/daemon.log filling up with kafka errors - https://phabricator.wikimedia.org/T187102 (10Ottomata) [15:43:44] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Data, 10Product-Infrastructure-Team-Backlog: Develop test environment solution for MEP analytics events - https://phabricator.wikimedia.org/T238837 (10Ottomata) [15:44:32] 10Analytics, 10Analytics-Kanban, 10MediaWiki-Vagrant: Vagrant's /var/log/daemon.log filling up with kafka errors - https://phabricator.wikimedia.org/T187102 (10Ottomata) In T238837, I'd like to make the default vagrant EventLogging setup not require Kafka. Hopefully that will at least work around this probl... [15:44:56] 10Analytics, 10Event-Platform, 10MediaWiki-Vagrant: Vagrant's /var/log/daemon.log filling up with kafka errors - https://phabricator.wikimedia.org/T187102 (10Ottomata) [15:45:22] 10Analytics, 10Event-Platform, 10MediaWiki-Vagrant: Vagrant's /var/log/daemon.log filling up with kafka errors - https://phabricator.wikimedia.org/T187102 (10DLynch) Makes sense to me. 👍🏻 [15:46:19] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Set up environment for Product Analytics system user - https://phabricator.wikimedia.org/T258970 (10Ottomata) a:03elukey [15:50:42] mforns: [15:50:42] https://meet.google.com/ntg-szxc-xam [15:56:55] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services): Set up Jenkins maven release job for wikimedia-event-utilities like analytics/refinery/source - https://phabricator.wikimedia.org/T259898 (10Ottomata) a:05Ottomata→03... [16:22:59] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Infrastructure-Data: session_tick stream configs - https://phabricator.wikimedia.org/T256311 (10mpopov) [16:23:44] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Infrastructure-Data: session_tick stream configs - https://phabricator.wikimedia.org/T256311 (10kzimmerman) @jlinehan @sdkim I'm unassigning @mpopov from this and we've moved this to Tracking on our board; please let me know if we shou... [16:25:19] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Infrastructure-Data: session_tick stream configs - https://phabricator.wikimedia.org/T256311 (10mpopov) [16:26:20] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Infrastructure-Data: session_tick stream configs - https://phabricator.wikimedia.org/T256311 (10mpopov) a:05mpopov→03None [16:27:35] 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog, 10Epic: [EPIC] Count unique iOS & Android users precisely and in a privacy conscious manner that does not require opt in to send data - https://phabricator.wikimedia.org/T202664 (10LGoto) p:05Triage→0... [16:42:07] 10Analytics-Radar, 10Product-Analytics, 10Structured-Data-Backlog: Create a Commons equivalent of the wikidata_entity table in the Data Lake - https://phabricator.wikimedia.org/T258834 (10LGoto) p:05Medium→03Low [16:50:24] 10Analytics-Radar, 10Discovery, 10Discovery-Analysis, 10Discovery-Search: UDF for language detection - https://phabricator.wikimedia.org/T182352 (10mpopov) [16:59:15] Hi a-team; is there a way to query a list wikis sorted by languages by size? Thanks. [17:00:14] hauskatze: what do you mean by size? [17:02:01] fdans: number of articles/content [17:02:47] I'm reviewing a patch for Pywikibot.git that requires listing the dbnames not in alphabetical order, but by size of the wiki [17:03:19] I'm looking at stats.wikimedia.org but I found nothing about it. Probably I am looking in the wrong place? [17:03:25] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [17:04:31] hauskatze: yes but wait, from the public APIs or the internal cluster we run? [17:04:54] hauskatze: if 2019 data is fine, there's fdans.com/wikigrowth [17:05:17] fdans: I'm afraid it's a new wiki [17:06:12] hauskatze: if it’s so new it’s not imported through our monthly sqoop then it’ll be a little trickier but I can show you how to query for most definitions of size [17:07:08] basically, the wmf.mediawiki_page_history table is one way and wmf_raw.mediawiki_page is another [17:07:43] milimetric: I shall see when it got created [17:07:44] checking [17:08:13] fwiw this is relevant for https://gerrit.wikimedia.org/r/c/pywikibot/core/+/620735 [17:11:20] 10Analytics-Radar, 10Performance-Team, 10MW-1.36-notes (1.36.0-wmf.4; 2020-08-11): Invalid navigation timing events - https://phabricator.wikimedia.org/T254606 (10Majavah) [17:12:13] 10Analytics-Clusters, 10Discovery, 10Discovery-Search (Current work): mjolnir-kafka-msearch-daemon dropping produced messages after move to search-loader[12]001 - https://phabricator.wikimedia.org/T260305 (10CBogen) [17:28:39] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Performance-Team: Parse user agents in navtiming instead of relying on eventlogging to do it - https://phabricator.wikimedia.org/T260580 (10Ottomata) [17:37:01] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Infrastructure-Team-Backlog, 10Epic: Vertical: Virtualpageview datastream on MEP - https://phabricator.wikimedia.org/T238138 (10Ottomata) [17:37:03] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10Product-Infrastructure-Data: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [17:37:05] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, and 2 others: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230 (10Ottomata) [17:47:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10Product-Infrastructure-Data: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [17:52:25] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Migrate EventLogging MediaViewer data to Event Platform - https://phabricator.wikimedia.org/T260582 (10Ottomata) [17:52:48] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10Product-Infrastructure-Data: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) [18:00:54] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10Product-Infrastructure-Data: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10jlinehan) Thanks @Ottomata this is great! I'd like to set something up this week to go down the list... [18:13:46] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10Product-Infrastructure-Data: Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata) Sounds great! I mostly did this so I could find one or two easy ones to start doing. I j... [18:15:34] mforns: [18:15:34] oh [18:15:35] https://gerrit.wikimedia.org/r/c/integration/config/+/618998 [18:15:40] sync up with hashar! [18:15:47] i guess he started working on it a tiny bit? [18:39:01] ottomata: oh! ok [18:58:10] (03PS1) 10Mforns: Tune threshold for hourly traffic anomaly detection [analytics/refinery] - 10https://gerrit.wikimedia.org/r/620767 (https://phabricator.wikimedia.org/T251814) [19:02:11] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Tune up thresholds of data quality hourly alarms - https://phabricator.wikimedia.org/T251814 (10mforns) As discussed in the last sync up meeting, I bumped up the threshold for hourly traffic anomaly detection to 15 times the standard deviation |p75 - media... [20:05:11] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Migrate EventLogging MediaViewer data to Event Platform - https://phabricator.wikimedia.org/T260582 (10Tgr) Any association I had with MediaViewer is lost in the mists of time. The Structured Data team is probably closest to being a maintainer (not sur... [20:05:48] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Migrate EventLogging MediaViewer data to Event Platform - https://phabricator.wikimedia.org/T260582 (10Ottomata) Oh if we can remove it that is even better! Who should we ping? [20:14:29] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Migrate EventLogging MediaViewer data to Event Platform - https://phabricator.wikimedia.org/T260582 (10Tgr) Someone would have to remove the logging code from MediaViewer first, I imagine? Or maybe disable it via site config, that's less effort. Accord... [20:17:06] (03PS3) 10Milimetric: [WIP] Spike so far [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/618874 (https://phabricator.wikimedia.org/T258532) [20:18:36] 10Analytics, 10Analytics-EventLogging, 10Event-Platform: Migrate EventLogging MediaViewer data to Event Platform - https://phabricator.wikimedia.org/T260582 (10Ottomata) K, trying to find pingables, from https://www.mediawiki.org/wiki/Reading/Multimedia/Structured_Data#Structured_Data_on_Wikimedia_Commons I... [20:20:37] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Spike so far [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/618874 (https://phabricator.wikimedia.org/T258532) (owner: 10Milimetric) [22:28:25] 10Analytics-Radar, 10MediaWiki-REST-API, 10Patch-For-Review, 10Platform Team Sprints Board (Sprint 1), 10Platform Team Workboards (Green): Unify access log schema for Action API and API Gateway/REST API - https://phabricator.wikimedia.org/T259736 (10Pchelolo) 05Open→03Resolved [22:28:28] 10Analytics, 10MediaWiki-REST-API, 10Patch-For-Review, 10Platform Team Sprints Board (Sprint 1), and 2 others: System administrator reviews API usage by client - https://phabricator.wikimedia.org/T251812 (10Pchelolo) [22:38:00] 10Analytics-Radar, 10Technical-blog-posts: Story idea for Blog: The Best Dataset on Wikimedia Content and Contributors - https://phabricator.wikimedia.org/T259559 (10srodlund) @Nuria I am back and happy to proceed! The note above says the post is "mostly" written. Let me know when you feel like it is done, an... [22:55:49] 10Analytics, 10MediaWiki-REST-API, 10Platform Team Sprints Board (Sprint 1), 10Platform Team Workboards (Green), 10Story: Enable TLS for api-gateway -> eventgate access log delivery - https://phabricator.wikimedia.org/T260626 (10Pchelolo) [22:56:46] 10Analytics, 10Event-Platform, 10Technical-blog-posts: Story idea for Blog: Wikimedia's Event Platform - https://phabricator.wikimedia.org/T253649 (10srodlund) @Ottomata I am back and will start editing these this week!