[00:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:19] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:29] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:00:31] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:38] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:01:09] 06SRE, 06Collaboration-Services: Document current expected SSH fingerprint for releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T443537#12529345 (10Dzahn) 05In progress→03Resolved ● **Updated both fingerprint pages** on Wikitech, they now match the current host keys: https://wikitech.wikim... [00:02:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [00:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:15:30] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.24 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:39:11] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1366942 [00:39:11] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1366942 (owner: 10TrainBranchBot) [00:45:17] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1366942 (owner: 10TrainBranchBot) [00:52:25] (03PS1) 10Jasmine: sophroid: [citoid prototype] Add resource_catalog and listeners to configmap and deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366949 (https://phabricator.wikimedia.org/T440259) [00:58:22] !log rzl@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply [00:58:30] !log rzl@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply [01:09:15] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1366967 [01:09:15] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1366967 (owner: 10TrainBranchBot) [01:09:39] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12529406 (10ArielGlenn) Perhaps the information from this task T428262 can be reused? Except that the relevant manager would be @MSantos for the MW release workin... [01:11:41] (03PS2) 10Jasmine: sophroid: [citoid prototype] Add resource_catalog and listeners to configmap and deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366949 (https://phabricator.wikimedia.org/T440259) [01:13:27] (03CR) 10RLazarus: [C:03+1] sophroid: [citoid prototype] Add resource_catalog and listeners to configmap and deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366949 (https://phabricator.wikimedia.org/T440259) (owner: 10Jasmine) [01:13:55] (03CR) 10Jasmine: [C:03+2] sophroid: [citoid prototype] Add resource_catalog and listeners to configmap and deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366949 (https://phabricator.wikimedia.org/T440259) (owner: 10Jasmine) [01:15:21] (03Merged) 10jenkins-bot: sophroid: [citoid prototype] Add resource_catalog and listeners to configmap and deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1366949 (https://phabricator.wikimedia.org/T440259) (owner: 10Jasmine) [01:15:28] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1366967 (owner: 10TrainBranchBot) [01:16:10] 06SRE, 06Collaboration-Services: Document current expected SSH fingerprint for releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T443537#12529424 (10matmarex) Thanks! I must have accepted the new key for releases1003 at some point without checking it against the doc pages, that's why I wasn't ge... [01:18:53] !log jasmine@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply [01:19:03] !log jasmine@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply [01:22:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [01:25:11] FIRING: Temperature: Temp issue on wdqs2023:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=wdqs2023 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [01:27:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [01:30:11] RESOLVED: Temperature: Temp issue on wdqs2023:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=wdqs2023 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [01:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.28% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [01:58:43] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367027 [02:00:44] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:06:09] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1367031 [02:08:29] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 07m 45s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:04] (03PS1) 10Andrew Bogott: Update instructions for restoring a backed up capi worker [puppet] - 10https://gerrit.wikimedia.org/r/1367060 (https://phabricator.wikimedia.org/T440363) [02:21:07] (03PS1) 10Andrew Bogott: magnum: refresh services when capi worker config is updated [puppet] - 10https://gerrit.wikimedia.org/r/1367061 (https://phabricator.wikimedia.org/T440363) [02:22:38] (03CR) 10Andrew Bogott: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1367061 (https://phabricator.wikimedia.org/T440363) (owner: 10Andrew Bogott) [02:30:35] (03CR) 10Andrew Bogott: [C:03+2] Update instructions for restoring a backed up capi worker [puppet] - 10https://gerrit.wikimedia.org/r/1367060 (https://phabricator.wikimedia.org/T440363) (owner: 10Andrew Bogott) [02:30:36] (03CR) 10Andrew Bogott: [C:03+2] magnum: refresh services when capi worker config is updated [puppet] - 10https://gerrit.wikimedia.org/r/1367061 (https://phabricator.wikimedia.org/T440363) (owner: 10Andrew Bogott) [04:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:34:46] 10SRE-swift-storage, 06Commons: Commons file not found - File:CitiusTech.JPG - https://phabricator.wikimedia.org/T443991#12529564 (10Peachey88) [05:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.32% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [06:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:29:42] (03PS3) 10Hashar: utils: vcl_ec2_nets: set shebang to python3 [puppet] - 10https://gerrit.wikimedia.org/r/1364492 (https://phabricator.wikimedia.org/T443798) [07:57:21] (03PS5) 10Hashar: logstash: change Object#kind_of? to Object#is_a? [puppet] - 10https://gerrit.wikimedia.org/r/1348957 (https://phabricator.wikimedia.org/T441847) [07:57:28] (03PS4) 10Hashar: jupyterhub: change Object#kind_of? to Object#is_a? [puppet] - 10https://gerrit.wikimedia.org/r/1348958 (https://phabricator.wikimedia.org/T441847) [08:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:30:03] FIRING: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:35:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.34% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [10:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:19:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12529724 (10phaultfinder) [11:44:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12529730 (10phaultfinder) [12:00:05] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:15] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:19] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:29] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:32] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:39] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:20:34] FIRING: DiskSpace: Disk space maps1011:9100:/srv 3.892% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=maps1011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [12:54:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12529791 (10phaultfinder) [13:07:26] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12529795 (10Dzahn) [13:10:07] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12529796 (10Dzahn) [13:11:09] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12529798 (10Dzahn) I copied the template for access requests to the task (what Denisse asked about). Otto already has existing shell access with other groups fr... [13:24:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12529829 (10phaultfinder) [13:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.44% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [14:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [14:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [15:15:34] RESOLVED: DiskSpace: Disk space maps1011:9100:/srv 0.01872% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=maps1011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [16:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:48:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.43% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [18:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:33:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:33:15] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [18:38:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:45:21] (03Abandoned) 10Krinkle: [WIP] varnish: Write tests for `?download` [puppet] - 10https://gerrit.wikimedia.org/r/1359400 (owner: 10Krinkle) [18:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [18:49:48] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:51:33] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:21:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:26:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:39:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12530227 (10phaultfinder) [21:14:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:19:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:25:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:30:22] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:49:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12530261 (10phaultfinder) [21:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.43% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [22:06:20] (03CR) 10Gehel: [C:03+1] Convert access for anakanishi from shell to LDAP [puppet] - 10https://gerrit.wikimedia.org/r/1366193 (https://phabricator.wikimedia.org/T442756) (owner: 10Muehlenhoff) [22:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:17:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:19:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12530263 (10phaultfinder) [22:22:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:25:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:30:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:47:06] FIRING: OsmSynchronisationLag: Maps - OSM synchronization lag - eqiad - https://wikitech.wikimedia.org/wiki/Maps/Runbook - https://grafana.wikimedia.org/d/000000305/maps-performances - https://alerts.wikimedia.org/?q=alertname%3DOsmSynchronisationLag [22:51:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:56:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:09:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12530269 (10phaultfinder) [23:25:18] (03PS1) 10Medelius: VisualEditor: add image for Suggestion Mode welcome dialog [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1367522 (https://phabricator.wikimedia.org/T442024) [23:35:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:40:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures