[00:52:37] FIRING: SystemdUnitFailed: wmf_auto_restart_sfacctd.service on netflow7002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:22:37] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [03:27:37] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:52:37] FIRING: SystemdUnitFailed: wmf_auto_restart_sfacctd.service on netflow7002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:57:37] RESOLVED: SystemdUnitFailed: wmf_auto_restart_sfacctd.service on netflow7002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:22:45] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [07:27:37] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [08:25:58] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: SSD firmware update not working in firmware cookbook - https://phabricator.wikimedia.org/T394543#10890078 (10Volans) Thanks @RKemper for the depool, I've performed the final run with the current PS in gerrit with test-cookbook for `ci... [08:56:38] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: SSD firmware update not working in firmware cookbook - https://phabricator.wikimedia.org/T394543#10890120 (10Volans) Forgot to mention, this is what I used to upgrade just the SSD firmware: `cookbook sre.hardware.upgrade-firmware -c... [11:10:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [11:22:37] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [11:45:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [12:24:40] 10netops, 06Infrastructure-Foundations, 06SRE: Errors for ifup@ens5.service after rebooting Ganeti VMs - https://phabricator.wikimedia.org/T273026#10890719 (10fgiunchedi) a:05fgiunchedi→03None [13:37:32] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp - https://phabricator.wikimedia.org/T396218#10890942 (10Reedy) [13:37:37] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from - https://phabricator.wikimedia.org/T396218#10890945 (10Reedy) p:05Triage→03Low [13:37:59] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10890947 (10Reedy) [13:38:45] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10890948 (10Reedy) Chrome and Safari on the device seem to behave the same… iOS is up to date on 18.5 [13:52:09] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891004 (10thcipriani) I see this 4× in the logs, it's (almost) all I see when I `grep -i reedy /var/log/gerrit/error_log` (the other one is a query timeout), guessing this... [13:55:55] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891020 (10MoritzMuehlenhoff) JFTR, Gerrit doesn't use CAS [14:02:20] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891049 (10SLyngshede-WMF) Quick test on iOS 18.5: Both Gerrit and IDP works fine. [14:05:16] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891073 (10SLyngshede-WMF) @Reedy Can you sign in to https://idp.wikimedia.org ? [14:05:17] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891074 (10Reedy) They’re fine on my phone which is also 18.5… Which I’m not sure if it makes it more or less odd. At least Tyler is seeing something on the logs… >>! In... [14:09:43] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891083 (10Reedy) >>! In T396218#10891073, @SLyngshede-WMF wrote: > @Reedy Can you sign in to https://idp.wikimedia.org ? Not on the iPad… fine on my iPhone. Phab LDAP lo... [14:09:48] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891086 (10SLyngshede-WMF) I don't have a current iPad to test on, but iPad OS 18.3.2 works fine. [14:10:56] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891092 (10SLyngshede-WMF) @Reedy Can you check the password in 1Password? Because my guess is that it's wrong in some way... or the username is :-) [14:15:24] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891126 (10Reedy) They both show identically on 1Password with the same last updated time… [14:18:45] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891136 (10SLyngshede-WMF) I wonder if 1Password is somehow broken in the latest iPadOS? Does it work if you manually type in the password? [14:20:13] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891138 (10SLyngshede-WMF) Quick search indicates that at least OTP tokens broken in 1Password with 18.5. [14:21:31] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891152 (10Reedy) It… kinda looks like when I get the window to choose which password, it’s not actually picking the one I’m selecting based on the length of the text… [14:22:59] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891158 (10SLyngshede-WMF) Could also be that the Apple Password manager somehow has an idea as to what the password might be and overwrites 1Password. [14:25:42] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891162 (10Reedy) Nope, nothing saved for Wikimedia sites there… [14:32:54] 10CAS-SSO, 10Gerrit, 06Infrastructure-Foundations: Unable to login to Gerrit/idp from iPad - https://phabricator.wikimedia.org/T396218#10891187 (10Reedy) 05Open→03Invalid It’s like some weird off by one error… I’ve updated that entry in my password manager to list domains more specifically. That see... [15:22:37] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [17:32:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [17:52:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [18:09:29] FIRING: SystemdUnitCrashLoop: node-bgpalerter.service crashloop on rpki1001:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [18:14:29] RESOLVED: SystemdUnitCrashLoop: node-bgpalerter.service crashloop on rpki1001:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [18:15:28] FIRING: SystemdUnitCrashLoop: node-bgpalerter.service crashloop on rpki1001:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [18:20:28] RESOLVED: SystemdUnitCrashLoop: node-bgpalerter.service crashloop on rpki1001:9100 - TODO - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitCrashLoop [19:22:45] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts [23:22:42] FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts