[00:45:28] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for aswikiquote - https://phabricator.wikimedia.org/T326885 (10Dcljr) This is a duplicate of {T321294}. [01:02:12] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Dreamy_Jazz) [02:30:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp1081 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp1081%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [02:35:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp1081 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp1081%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:21:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp1080 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp1080%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [03:26:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp1080 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp1080%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:25:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp1082 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp1082%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:30:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp1082 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp1082%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:55:13] PROBLEM - Webrequests Varnishkafka log producer on cp1084 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.64.32.68: Connection reset by peer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [05:07:29] PROBLEM - Webrequests Varnishkafka log producer on cp1084 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [05:10:01] RECOVERY - Webrequests Varnishkafka log producer on cp1084 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [07:02:28] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for aswikiquote - https://phabricator.wikimedia.org/T326885 (10Marostegui) Correct, the views are present already [07:02:59] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for aswikiquote - https://phabricator.wikimedia.org/T321294 (10Marostegui) [07:03:04] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for aswikiquote - https://phabricator.wikimedia.org/T326885 (10Marostegui) [08:01:10] 10Data-Engineering-Planning, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10Marostegui) [10:03:02] 10Data-Engineering, 10SRE, 10Shared-Data-Infrastructure: geoip_update_main failure on puppetmaster1001 - https://phabricator.wikimedia.org/T324548 (10BTullis) 05Open→03Resolved As far as I am aware, we don't actually need to change any files on the puppetmaster(s) when the Maxmind licence is renewed. The... [10:11:33] !log roll-restart aqs to update mediawiki_history_snapshot to 2023-01 [10:11:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:53:26] Is https://gerrit.wikimedia.org/g/analytics/geowiki/+/9a0e7187da7508fb314340a61427b19a340e2635/geowiki/mysql_config.py still being used? [10:56:09] 10Data-Engineering-Planning, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10MoritzMuehlenhoff) [11:45:41] zabe: I suspect it's not used any more, based on my reading of this: https://wikitech.wikimedia.org/wiki/Analytics/Archive/Geowiki - but I' sure other people (e.g. milimetric) will know for sure. [12:03:25] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Wikidata: Realtime Wikibase editing UI and API - https://phabricator.wikimedia.org/T298305 (10Lydia_Pintscher) 05Open→03Declined Realistically this is not something we can put time into. Sorry. There are just too many other things that are mor... [12:57:14] (03PS1) 10Kosta Harlan: homepagemodule: Document daily_total_views_count in action_data [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886355 (https://phabricator.wikimedia.org/T328391) [13:21:38] Ben's correct, zabe, geowiki is the old name of geoeditors, so we don't use it anymore. I marked that repo read only as it looks like we forgot to do that at the time, sorry for the noise. [14:13:36] no problem, just wanted to make since the cu query there would have been outdated then due to a migration i'm doing, thanks :) [14:17:47] btullis: o/ I created https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/886317 to hopefully automate the steps to reimage all nodes in a k8s cluster, I'll wait for serviceops' review next week and then test it on ml-staging-codfw [14:19:40] Woah! Excellent work. I was just checking out the ganeti reimage cookbook, which is great, but this is even greater. [14:20:36] thanks! The only bit that may be slow is the fact that workers are reimaged one at the time, I could probably add some parallelism (maybe 2 at the time or similar) [14:21:08] dse and ml-serve clusters have 8 worker nodes and it may take a bit [14:26:03] 10Data-Engineering, 10Event-Platform Value Stream: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment - https://phabricator.wikimedia.org/T328478 (10dcausse) >>! In T328478#8583469, @gmodena wrote: >>>! In T328478#8581389, @dcausse wrote: >> I think it's important for the `flink-... [14:28:27] 10Data-Engineering-Planning, 10Cassandra, 10Image-Suggestions, 10Section-Level-Image-Suggestions: Section Level Image Suggestions - Data Persistence Request - https://phabricator.wikimedia.org/T320831 (10mfossati) [14:54:48] (03PS3) 10Mazevedo: Add MobileWikiAppiOSLoginAction schema to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886141 (https://phabricator.wikimedia.org/T328697) [15:18:42] (03PS4) 10Mazevedo: Add MobileWikiAppiOSLoginAction schema to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886141 (https://phabricator.wikimedia.org/T328697) [15:23:33] !log deployed airflow-dags/analytics to disable skein log collection from the SparkSubmitOperator. [15:23:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:24:26] note: all other instances deployed from now on will also disable log collection ^ (as the change was in wmf_common). I'm not sure if we have processes to let people know of changes like this, but it feels like a safe change given that anytime we leave log collection on we're probably just forgetting to set it to False. [15:24:40] 10Data-Engineering-Planning, 10Data Pipelines, 10Pageviews-Anomaly, 10Product-Analytics, and 2 others: Analyze possible bot traffic for frwiki article Cookie (informatique) - https://phabricator.wikimedia.org/T313114 (10hashar) Last time I have checked in most of the traffic to https://fr.wikipedia.org/wik... [15:26:57] (03PS4) 10Mazevedo: Add legacy schema MobileWikiAppiOSReadingLists to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885835 (https://phabricator.wikimedia.org/T328487) [15:29:11] (03PS5) 10Mazevedo: Add legacy schema MobileWikiAppiOSReadingLists to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885835 (https://phabricator.wikimedia.org/T328487) [15:37:39] 10Data-Engineering-Planning, 10Cassandra, 10Image-Suggestions, 10Section-Level-Image-Suggestions: Section Level Image Suggestions - Data Persistence Request - https://phabricator.wikimedia.org/T320831 (10mfossati) >>! In T320831#8583283, @Eevans wrote: > We have a cluster that exists for testing, //for som... [15:37:51] 10Data-Engineering-Planning, 10Cassandra, 10Image-Suggestions, 10Section-Level-Image-Suggestions: Section Level Image Suggestions - Data Persistence Request - https://phabricator.wikimedia.org/T320831 (10mfossati) [16:18:12] 10Data-Engineering-Planning, 10Data-Catalog, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Datahub user records are not being created after login - https://phabricator.wikimedia.org/T327884 (10BTullis) Hi @Stevemunene did this get deployed in the end? If so, I guess it didn't work because I sti... [17:04:44] (03CR) 10Mazevedo: Add MobileWikiAppiOSLoginAction schema to MEP (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886141 (https://phabricator.wikimedia.org/T328697) (owner: 10Mazevedo) [17:17:23] 10Data-Engineering-Planning, 10Data-Catalog, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)): Datahub user records are not being created after login - https://phabricator.wikimedia.org/T327884 (10Stevemunene) Hi, This was deployed and tested on staging environment and the records were still not cre... [17:27:43] (03CR) 10Tsevener: [C: 04-1] "Thanks! The login_action files good - can you double-check the others? I don't think they should be showing as whole added files and rathe" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886141 (https://phabricator.wikimedia.org/T328697) (owner: 10Mazevedo) [17:41:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp1089 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp1089%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:46:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp1089 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp1089%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [18:05:18] (03CR) 10Mazevedo: Add MobileWikiAppiOSLoginAction schema to MEP (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886141 (https://phabricator.wikimedia.org/T328697) (owner: 10Mazevedo) [18:19:47] (03CR) 10Tsevener: [C: 03+2] Add MobileWikiAppiOSLoginAction schema to MEP (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886141 (https://phabricator.wikimedia.org/T328697) (owner: 10Mazevedo) [18:20:19] (03Merged) 10jenkins-bot: Add MobileWikiAppiOSLoginAction schema to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886141 (https://phabricator.wikimedia.org/T328697) (owner: 10Mazevedo) [18:24:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp1088 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp1088%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [18:29:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp1088 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp1088%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [19:26:58] (03PS2) 10Kosta Harlan: homepagemodule: Document total_pageviews_count in action_data [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/886355 (https://phabricator.wikimedia.org/T328391) [19:46:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp1090 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp1090%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [19:51:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp1090 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqiad%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp1090%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [20:16:50] 10Data-Engineering-Planning, 10Data Pipelines, 10Pageviews-Anomaly, 10Product-Analytics, and 2 others: Analyze possible bot traffic for frwiki article Cookie (informatique) - https://phabricator.wikimedia.org/T313114 (10kzimmerman) @PBradley-WMF I took a look at the top linking sites for https://fr.wikiped... [20:27:59] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Dreamy_Jazz) [20:45:17] (03CR) 10Tsevener: [C: 03+2] Add legacy schema MobileWikiAppiOSReadingLists to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885835 (https://phabricator.wikimedia.org/T328487) (owner: 10Mazevedo) [20:45:46] (03Merged) 10jenkins-bot: Add legacy schema MobileWikiAppiOSReadingLists to MEP [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/885835 (https://phabricator.wikimedia.org/T328487) (owner: 10Mazevedo) [21:47:48] 10Analytics-Radar, 10Data-Persistence (work done), 10Platform Engineering Roadmap Decision Making, 10Epic, and 5 others: Remove revision_comment_temp and revision_actor_temp - https://phabricator.wikimedia.org/T215466 (10Zabe) [22:14:54] 10Data-Engineering-Planning, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10colewhite) [23:09:45] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Zabe)