[08:48:48] federico3: I am not sure how to tackle this alert [09:47:54] FIRING: ZarcilloDataIngestionLag: Zarcillo data ingestion lag too high for table datacenters - https://wikitech.wikimedia.org/wiki/MariaDB/Zarcillo - https://grafana.wikimedia.org/d/celrzpf6av8qob/zarcillo - https://alerts.wikimedia.org/?q=alertname%3DZarcilloDataIngestionLag [08:49:32] looks like it started last night at around 9pm [08:52:18] "httpx.ConnectTimeout: timed out" [08:53:09] That looks to be due to etcd [08:57:05] was it the DC switchover? [08:57:56] federico3: only services yesterday [08:58:04] effie: ^ right? [08:58:43] I suspect https://gitlab.wikimedia.org/repos/data_persistence/zarcillo/-/blob/main/zarcillo/api/datasync.py#L93 [08:59:02] It would help if there were timestamps in the log messages, federico3 is there a reason that they are not present? [08:59:47] if you do SELECT * FROM datacenters; you should see the old values, I suspect the datacenter yaml changed [08:59:47] federico3: today we are switching MW too [09:00:12] the logs have timestamps in ECS format [09:00:37] The ones that you get when tailing [09:00:50] aha: https://config-master.wikimedia.org/mediawiki.yaml [09:01:55] indeed both DCs are flagged as not read only which confused https://gitlab.wikimedia.org/repos/data_persistence/zarcillo/-/blob/main/zarcillo/api/datasync.py#L107 [09:02:20] as soon as the yaml goes back to active/standby the warning will clear [09:02:46] in the meantime the only effect is the stale data for now [09:03:17] federico3: right, so I guess today/tomorrow [09:03:40] federico3: what can be done to prevent these situations? any idea? [09:03:44] yep - the active/standby data is used only to color the rows in the tables [09:06:24] yes, tweak the code I linked to support both dc being RW together (pretty easy) and look around the codebase where the table is used, e.g. https://gitlab.wikimedia.org/repos/data_persistence/zarcillo/-/blob/main/zarcillo/api/main.py#L133 so that during the next switchover the UI works fine [09:40:59] federico3: that function does not trigger an exception when running locally and I have the correct ingestion timestamps in the pod DB [09:41:19] That is from one commit ahead though [09:43:54] select * from datacenters; on prod shows stale rows and yet both are set as RW so the double-RW part works - but somehow it stopped updating it after that [09:49:16] aha [09:49:42] https://usercontent.irccloud-cdn.com/file/DcSHiB0O/image.png [09:50:58] datasync.py:96 [cmmy] starting ingestion from https://config-master.discovery.wmnet/ followed by a timeout after few lines [10:15:19] marostegui: do you happen to know why db117[4-5] don't seem to be in dbctl? [10:16:00] yes: https://phabricator.wikimedia.org/T441800 and https://phabricator.wikimedia.org/T436904 [10:18:03] +1 - I will create a ticket so that there is a way to distinguish that in Zarcillo rather than just "depooled" [10:18:57] the transition in between removed from dbctl and decommissioned usually takes way less, but due to $lotsofotherthings I've had them sitting for longer than usual [10:23:05] https://phabricator.wikimedia.org/T442685 [10:25:04] do you see the timeout above? notice the .discovery.wmnet endpoint is being used from k8s [10:33:49] federico3: yes, that responds from cumin though [10:43:44] It looks like exec is blocked, else I was going to call socket.gethostbyname to see if that produced an issue [11:02:16] federico3: given that other calls go to wikimedia.org (e.g. netbox etc) is there a reason that config-master requests instead go to the internal name? [11:03:06] ah, nvm I found the internal names too [11:03:31] They are working though as netbox just passed by in the logs [11:19:25] FIRING: SystemdUnitFailed: swift_ring_manager.service on ms-fe1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:22:53] huh [11:23:50] [swift-dispersion-report timed out] [11:27:57] re-run went OK. [11:29:25] RESOLVED: SystemdUnitFailed: swift_ring_manager.service on ms-fe1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:49:42] FIRING: [2x] ProbeDown: Service dborch1002:443 has failed probes (http_orchestrator_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Orchestrator - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:02:20] RESOLVED: [2x] ProbeDown: Service dborch1002:443 has failed probes (http_orchestrator_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Orchestrator - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:12:26] FIRING: [6x] MysqlReplicationLagPtHeartbeat: MySQL instance db1901:9104 has too large replication lag (11m 22s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [12:13:56] RESOLVED: [6x] MysqlReplicationLagPtHeartbeat: MySQL instance db1901:9104 has too large replication lag (15m 22s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [12:54:21] It data-persistence still OK with the DC switchover? [13:22:51] slyngs: well, the clusters all look fine in orchestrator, so I believe so. Not sure if marostegui has any concerns [13:23:35] Thank you, I'll check again before starting [14:04:13] sigh, my pontoon stack has bitrotted into a disaster. Again. [14:50:37] marostegui: Any concerns about me doing the switch-over in ten minutes? [14:53:20] slyngs: no, but I'm not going to be actively online unless something breaks. I've been up since 5am for on-call [14:53:34] If something breaks please call me [14:53:45] Will do, thank you. [16:14:01] q: should https://noc.wikimedia.org/db.php default to the primary datacenter? [16:15:58] yeah, I guess noone had time to implement it. [16:40:45] Amir1: thanks, so long as it is expected that it doesn't right now, that answers my question!