[00:05:37] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24753 and previous config saved to /var/cache/conftool/dbconfig/20220417-000536-ladsgroup.json [00:05:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:09:55] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24754 and previous config saved to /var/cache/conftool/dbconfig/20220417-000954-ladsgroup.json [00:09:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:20:42] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24755 and previous config saved to /var/cache/conftool/dbconfig/20220417-002041-ladsgroup.json [00:20:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:25:00] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24756 and previous config saved to /var/cache/conftool/dbconfig/20220417-002459-ladsgroup.json [00:25:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:35:47] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24757 and previous config saved to /var/cache/conftool/dbconfig/20220417-003546-ladsgroup.json [00:35:48] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [00:35:50] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [00:35:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:35:52] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [00:35:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:35:55] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24758 and previous config saved to /var/cache/conftool/dbconfig/20220417-003554-ladsgroup.json [00:35:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:35:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:37:13] (KubernetesRsyslogDown) firing: rsyslog on kubernetes1018:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [00:40:05] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24759 and previous config saved to /var/cache/conftool/dbconfig/20220417-004004-ladsgroup.json [00:40:07] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance [00:40:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:40:08] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance [00:40:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:40:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:40:13] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24760 and previous config saved to /var/cache/conftool/dbconfig/20220417-004013-ladsgroup.json [00:40:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:51:50] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24761 and previous config saved to /var/cache/conftool/dbconfig/20220417-005150-ladsgroup.json [00:51:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:51:55] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [01:06:55] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24762 and previous config saved to /var/cache/conftool/dbconfig/20220417-010655-ladsgroup.json [01:06:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:22:00] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24763 and previous config saved to /var/cache/conftool/dbconfig/20220417-012200-ladsgroup.json [01:22:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:36:10] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24764 and previous config saved to /var/cache/conftool/dbconfig/20220417-013609-ladsgroup.json [01:36:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:36:14] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [01:37:05] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24765 and previous config saved to /var/cache/conftool/dbconfig/20220417-013705-ladsgroup.json [01:37:07] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [01:37:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:37:09] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [01:37:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:37:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:37:14] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24766 and previous config saved to /var/cache/conftool/dbconfig/20220417-013713-ladsgroup.json [01:37:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:38:45] (JobUnavailable) firing: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:47:54] (NodeTextfileStale) firing: Stale textfile for cloudcontrol2001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [01:48:40] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24767 and previous config saved to /var/cache/conftool/dbconfig/20220417-014839-ladsgroup.json [01:48:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:48:44] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [01:48:45] (JobUnavailable) resolved: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:51:15] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24768 and previous config saved to /var/cache/conftool/dbconfig/20220417-015114-ladsgroup.json [01:51:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:03:45] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24769 and previous config saved to /var/cache/conftool/dbconfig/20220417-020344-ladsgroup.json [02:03:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:06:20] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24770 and previous config saved to /var/cache/conftool/dbconfig/20220417-020619-ladsgroup.json [02:06:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:18:50] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24771 and previous config saved to /var/cache/conftool/dbconfig/20220417-021849-ladsgroup.json [02:18:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:21:25] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24772 and previous config saved to /var/cache/conftool/dbconfig/20220417-022124-ladsgroup.json [02:21:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:21:29] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [02:21:33] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance [02:21:35] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance [02:21:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:21:36] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [02:21:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:21:39] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [02:21:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:21:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:21:44] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24773 and previous config saved to /var/cache/conftool/dbconfig/20220417-022143-ladsgroup.json [02:21:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:25:54] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24774 and previous config saved to /var/cache/conftool/dbconfig/20220417-022554-ladsgroup.json [02:25:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:32:55] (NodeTextfileStale) firing: (3) Stale textfile for elastic1075:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [02:33:55] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24775 and previous config saved to /var/cache/conftool/dbconfig/20220417-023354-ladsgroup.json [02:33:57] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [02:33:58] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [02:33:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:33:59] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [02:34:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:34:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:34:03] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24776 and previous config saved to /var/cache/conftool/dbconfig/20220417-023403-ladsgroup.json [02:34:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:40:59] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24777 and previous config saved to /var/cache/conftool/dbconfig/20220417-024059-ladsgroup.json [02:41:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:45:41] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24778 and previous config saved to /var/cache/conftool/dbconfig/20220417-024540-ladsgroup.json [02:45:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:45:45] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [02:56:04] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24779 and previous config saved to /var/cache/conftool/dbconfig/20220417-025604-ladsgroup.json [02:56:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:00:46] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24780 and previous config saved to /var/cache/conftool/dbconfig/20220417-030045-ladsgroup.json [03:00:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:01:55] (NodeTextfileStale) firing: Stale textfile for ms-be2067:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [03:11:09] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24781 and previous config saved to /var/cache/conftool/dbconfig/20220417-031109-ladsgroup.json [03:11:11] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance [03:11:12] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance [03:11:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:11:14] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [03:11:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:11:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:11:18] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24782 and previous config saved to /var/cache/conftool/dbconfig/20220417-031117-ladsgroup.json [03:11:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:15:51] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24783 and previous config saved to /var/cache/conftool/dbconfig/20220417-031551-ladsgroup.json [03:15:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:24:33] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24784 and previous config saved to /var/cache/conftool/dbconfig/20220417-032433-ladsgroup.json [03:24:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:24:38] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [03:30:56] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24785 and previous config saved to /var/cache/conftool/dbconfig/20220417-033056-ladsgroup.json [03:30:58] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance [03:30:59] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance [03:30:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:31:01] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [03:31:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:31:04] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24786 and previous config saved to /var/cache/conftool/dbconfig/20220417-033104-ladsgroup.json [03:31:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:31:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:39:38] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24787 and previous config saved to /var/cache/conftool/dbconfig/20220417-033938-ladsgroup.json [03:39:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:43:01] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24788 and previous config saved to /var/cache/conftool/dbconfig/20220417-034300-ladsgroup.json [03:43:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:43:05] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [03:54:44] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24789 and previous config saved to /var/cache/conftool/dbconfig/20220417-035443-ladsgroup.json [03:54:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:58:06] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24790 and previous config saved to /var/cache/conftool/dbconfig/20220417-035805-ladsgroup.json [03:58:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:01:38] RECOVERY - Check systemd state on build2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [04:09:49] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24791 and previous config saved to /var/cache/conftool/dbconfig/20220417-040948-ladsgroup.json [04:09:50] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [04:09:52] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [04:09:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:09:53] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [04:09:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:09:57] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24792 and previous config saved to /var/cache/conftool/dbconfig/20220417-040956-ladsgroup.json [04:09:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:10:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:10:38] PROBLEM - Check systemd state on build2001 is CRITICAL: CRITICAL - degraded: The following units failed: debian-weekly-rebuild.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [04:13:11] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24793 and previous config saved to /var/cache/conftool/dbconfig/20220417-041310-ladsgroup.json [04:13:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:21:30] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24794 and previous config saved to /var/cache/conftool/dbconfig/20220417-042129-ladsgroup.json [04:21:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:21:34] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [04:28:16] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24795 and previous config saved to /var/cache/conftool/dbconfig/20220417-042815-ladsgroup.json [04:28:17] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [04:28:19] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [04:28:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:28:20] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [04:28:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:28:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:36:35] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24796 and previous config saved to /var/cache/conftool/dbconfig/20220417-043634-ladsgroup.json [04:36:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:37:13] (KubernetesRsyslogDown) firing: rsyslog on kubernetes1018:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [04:38:00] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance [04:38:02] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance [04:38:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:38:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:47:43] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance [04:47:44] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance [04:47:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:47:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:51:40] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24797 and previous config saved to /var/cache/conftool/dbconfig/20220417-045139-ladsgroup.json [04:51:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:56:45] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [04:56:47] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [04:56:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:56:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:05:48] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance [05:05:49] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance [05:05:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:05:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:05:54] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24798 and previous config saved to /var/cache/conftool/dbconfig/20220417-050553-ladsgroup.json [05:05:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:05:58] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [05:06:45] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24799 and previous config saved to /var/cache/conftool/dbconfig/20220417-050644-ladsgroup.json [05:06:46] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance [05:06:48] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance [05:06:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:06:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:06:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:06:53] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24800 and previous config saved to /var/cache/conftool/dbconfig/20220417-050652-ladsgroup.json [05:06:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:18:32] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24801 and previous config saved to /var/cache/conftool/dbconfig/20220417-051831-ladsgroup.json [05:18:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:18:36] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [05:20:38] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24802 and previous config saved to /var/cache/conftool/dbconfig/20220417-052037-ladsgroup.json [05:20:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:33:37] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24803 and previous config saved to /var/cache/conftool/dbconfig/20220417-053336-ladsgroup.json [05:33:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:34:58] PROBLEM - BGP status on cr2-eqiad is CRITICAL: BGP CRITICAL - AS64605/IPv6: Active - Anycast https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status [05:35:43] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24804 and previous config saved to /var/cache/conftool/dbconfig/20220417-053542-ladsgroup.json [05:35:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:47:54] (NodeTextfileStale) firing: Stale textfile for cloudcontrol2001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [05:48:42] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P24805 and previous config saved to /var/cache/conftool/dbconfig/20220417-054841-ladsgroup.json [05:48:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:50:48] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24806 and previous config saved to /var/cache/conftool/dbconfig/20220417-055047-ladsgroup.json [05:50:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:03:47] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24807 and previous config saved to /var/cache/conftool/dbconfig/20220417-060346-ladsgroup.json [06:03:48] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [06:03:50] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [06:03:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:03:54] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [06:03:55] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24808 and previous config saved to /var/cache/conftool/dbconfig/20220417-060354-ladsgroup.json [06:03:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:03:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:04:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:04:52] PROBLEM - SSH on furud.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [06:05:53] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24809 and previous config saved to /var/cache/conftool/dbconfig/20220417-060552-ladsgroup.json [06:05:55] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [06:05:56] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [06:05:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:06:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:06:01] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24810 and previous config saved to /var/cache/conftool/dbconfig/20220417-060600-ladsgroup.json [06:06:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:10:18] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24811 and previous config saved to /var/cache/conftool/dbconfig/20220417-061017-ladsgroup.json [06:10:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:10:22] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [06:15:15] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24812 and previous config saved to /var/cache/conftool/dbconfig/20220417-061514-ladsgroup.json [06:15:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:25:23] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24813 and previous config saved to /var/cache/conftool/dbconfig/20220417-062522-ladsgroup.json [06:25:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:30:20] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24814 and previous config saved to /var/cache/conftool/dbconfig/20220417-063019-ladsgroup.json [06:30:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:32:55] (NodeTextfileStale) firing: (3) Stale textfile for elastic1075:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [06:40:28] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24815 and previous config saved to /var/cache/conftool/dbconfig/20220417-064027-ladsgroup.json [06:40:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:45:25] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24816 and previous config saved to /var/cache/conftool/dbconfig/20220417-064524-ladsgroup.json [06:45:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:55:33] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24817 and previous config saved to /var/cache/conftool/dbconfig/20220417-065532-ladsgroup.json [06:55:34] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [06:55:36] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [06:55:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:55:37] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [06:55:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:55:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20220417T0700) [07:00:30] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24818 and previous config saved to /var/cache/conftool/dbconfig/20220417-070029-ladsgroup.json [07:00:31] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [07:00:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:00:33] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [07:00:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:00:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:00:38] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24819 and previous config saved to /var/cache/conftool/dbconfig/20220417-070037-ladsgroup.json [07:00:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:00:42] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [07:01:55] (NodeTextfileStale) firing: Stale textfile for ms-be2067:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [07:05:00] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance [07:05:02] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance [07:05:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:05:03] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance [07:05:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:05:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:05:09] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance [07:05:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:06:02] RECOVERY - SSH on furud.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [07:10:38] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24820 and previous config saved to /var/cache/conftool/dbconfig/20220417-071038-ladsgroup.json [07:10:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:10:42] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [07:14:32] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance [07:14:33] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance [07:14:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:14:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:24:19] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance [07:24:20] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance [07:24:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:24:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:24:25] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24821 and previous config saved to /var/cache/conftool/dbconfig/20220417-072425-ladsgroup.json [07:24:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:24:29] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [07:25:43] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24822 and previous config saved to /var/cache/conftool/dbconfig/20220417-072543-ladsgroup.json [07:25:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:40:48] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24823 and previous config saved to /var/cache/conftool/dbconfig/20220417-074048-ladsgroup.json [07:40:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:55:53] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24824 and previous config saved to /var/cache/conftool/dbconfig/20220417-075553-ladsgroup.json [07:55:55] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance [07:55:56] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance [07:55:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:55:58] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [07:56:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:56:01] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24825 and previous config saved to /var/cache/conftool/dbconfig/20220417-075601-ladsgroup.json [07:56:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:56:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:07:16] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24826 and previous config saved to /var/cache/conftool/dbconfig/20220417-080715-ladsgroup.json [08:07:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:07:20] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [08:22:21] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24827 and previous config saved to /var/cache/conftool/dbconfig/20220417-082220-ladsgroup.json [08:22:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:24:40] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24828 and previous config saved to /var/cache/conftool/dbconfig/20220417-082439-ladsgroup.json [08:24:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:24:44] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [08:37:13] (KubernetesRsyslogDown) firing: rsyslog on kubernetes1018:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [08:37:26] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24829 and previous config saved to /var/cache/conftool/dbconfig/20220417-083725-ladsgroup.json [08:37:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:39:45] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24830 and previous config saved to /var/cache/conftool/dbconfig/20220417-083944-ladsgroup.json [08:39:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:52:31] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24831 and previous config saved to /var/cache/conftool/dbconfig/20220417-085231-ladsgroup.json [08:52:33] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance [08:52:34] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance [08:52:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:52:36] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [08:52:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:52:39] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24832 and previous config saved to /var/cache/conftool/dbconfig/20220417-085239-ladsgroup.json [08:52:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:52:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:54:50] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24833 and previous config saved to /var/cache/conftool/dbconfig/20220417-085449-ladsgroup.json [08:54:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:04:15] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24834 and previous config saved to /var/cache/conftool/dbconfig/20220417-090414-ladsgroup.json [09:04:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:04:19] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [09:09:55] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24835 and previous config saved to /var/cache/conftool/dbconfig/20220417-090954-ladsgroup.json [09:09:56] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [09:09:58] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [09:09:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:10:02] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [09:10:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:10:04] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24836 and previous config saved to /var/cache/conftool/dbconfig/20220417-091002-ladsgroup.json [09:10:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:10:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:19:20] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24837 and previous config saved to /var/cache/conftool/dbconfig/20220417-091919-ladsgroup.json [09:19:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:34:25] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24838 and previous config saved to /var/cache/conftool/dbconfig/20220417-093424-ladsgroup.json [09:34:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:47:54] (NodeTextfileStale) firing: Stale textfile for cloudcontrol2001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:49:30] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24839 and previous config saved to /var/cache/conftool/dbconfig/20220417-094929-ladsgroup.json [09:49:31] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance [09:49:33] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance [09:49:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:49:34] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [09:49:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:49:38] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24840 and previous config saved to /var/cache/conftool/dbconfig/20220417-094937-ladsgroup.json [09:49:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:49:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:50:18] PROBLEM - SSH on wtp1045.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [10:02:04] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24841 and previous config saved to /var/cache/conftool/dbconfig/20220417-100203-ladsgroup.json [10:02:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:02:08] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [10:10:19] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24842 and previous config saved to /var/cache/conftool/dbconfig/20220417-101019-ladsgroup.json [10:10:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:10:25] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [10:17:09] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24843 and previous config saved to /var/cache/conftool/dbconfig/20220417-101708-ladsgroup.json [10:17:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:25:24] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24844 and previous config saved to /var/cache/conftool/dbconfig/20220417-102524-ladsgroup.json [10:25:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:32:14] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24845 and previous config saved to /var/cache/conftool/dbconfig/20220417-103213-ladsgroup.json [10:32:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:32:55] (NodeTextfileStale) firing: (3) Stale textfile for elastic1075:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [10:40:30] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24846 and previous config saved to /var/cache/conftool/dbconfig/20220417-104029-ladsgroup.json [10:40:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:47:19] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24847 and previous config saved to /var/cache/conftool/dbconfig/20220417-104718-ladsgroup.json [10:47:21] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance [10:47:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:47:22] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance [10:47:23] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [10:47:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:47:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:47:27] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24848 and previous config saved to /var/cache/conftool/dbconfig/20220417-104727-ladsgroup.json [10:47:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:55:35] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24849 and previous config saved to /var/cache/conftool/dbconfig/20220417-105534-ladsgroup.json [10:55:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:55:39] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [10:55:43] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance [10:55:44] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance [10:55:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:55:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:58:56] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24850 and previous config saved to /var/cache/conftool/dbconfig/20220417-105855-ladsgroup.json [10:58:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:01:55] (NodeTextfileStale) firing: Stale textfile for ms-be2067:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [11:05:16] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance [11:05:18] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2129.codfw.wmnet with reason: Maintenance [11:05:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:05:19] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on 8 hosts with reason: Maintenance [11:05:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:05:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:05:25] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 8 hosts with reason: Maintenance [11:05:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:14:01] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24851 and previous config saved to /var/cache/conftool/dbconfig/20220417-111400-ladsgroup.json [11:14:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:14:57] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [11:14:58] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [11:14:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:15:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:24:26] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [11:24:27] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [11:24:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:24:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:24:32] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24852 and previous config saved to /var/cache/conftool/dbconfig/20220417-112432-ladsgroup.json [11:24:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:24:36] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [11:28:54] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24853 and previous config saved to /var/cache/conftool/dbconfig/20220417-112854-ladsgroup.json [11:28:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:29:06] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24854 and previous config saved to /var/cache/conftool/dbconfig/20220417-112905-ladsgroup.json [11:29:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:33:06] PROBLEM - Varnish traffic drop between 30min ago and now at eqsin on alert1001 is CRITICAL: 55.2 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [11:34:14] PROBLEM - Varnish traffic drop between 30min ago and now at eqiad on alert1001 is CRITICAL: 40.91 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [11:35:24] RECOVERY - Varnish traffic drop between 30min ago and now at eqsin on alert1001 is OK: (C)60 le (W)70 le 87.22 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [11:36:32] RECOVERY - Varnish traffic drop between 30min ago and now at eqiad on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [11:43:59] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24855 and previous config saved to /var/cache/conftool/dbconfig/20220417-114359-ladsgroup.json [11:44:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:11] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24856 and previous config saved to /var/cache/conftool/dbconfig/20220417-114411-ladsgroup.json [11:44:13] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance [11:44:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:14] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance [11:44:15] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [11:44:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:19] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24857 and previous config saved to /var/cache/conftool/dbconfig/20220417-114419-ladsgroup.json [11:44:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:52:36] RECOVERY - SSH on wtp1045.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [11:55:42] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24858 and previous config saved to /var/cache/conftool/dbconfig/20220417-115541-ladsgroup.json [11:55:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:55:46] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [11:59:05] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24859 and previous config saved to /var/cache/conftool/dbconfig/20220417-115904-ladsgroup.json [11:59:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:10:47] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24860 and previous config saved to /var/cache/conftool/dbconfig/20220417-121046-ladsgroup.json [12:10:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:14:10] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24861 and previous config saved to /var/cache/conftool/dbconfig/20220417-121409-ladsgroup.json [12:14:11] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance [12:14:13] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance [12:14:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:14:14] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [12:14:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:14:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:14:18] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24862 and previous config saved to /var/cache/conftool/dbconfig/20220417-121417-ladsgroup.json [12:14:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:25:52] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24863 and previous config saved to /var/cache/conftool/dbconfig/20220417-122551-ladsgroup.json [12:25:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:26:20] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24864 and previous config saved to /var/cache/conftool/dbconfig/20220417-122619-ladsgroup.json [12:26:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:26:24] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [12:37:13] (KubernetesRsyslogDown) firing: rsyslog on kubernetes1018:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [12:40:57] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24865 and previous config saved to /var/cache/conftool/dbconfig/20220417-124056-ladsgroup.json [12:40:59] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance [12:41:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:41:01] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance [12:41:01] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [12:41:02] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [12:41:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:41:05] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [12:41:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:41:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:41:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:41:10] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24866 and previous config saved to /var/cache/conftool/dbconfig/20220417-124109-ladsgroup.json [12:41:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:41:25] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24867 and previous config saved to /var/cache/conftool/dbconfig/20220417-124125-ladsgroup.json [12:41:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:53:46] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24868 and previous config saved to /var/cache/conftool/dbconfig/20220417-125346-ladsgroup.json [12:53:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:53:50] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [12:56:30] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24869 and previous config saved to /var/cache/conftool/dbconfig/20220417-125630-ladsgroup.json [12:56:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:08:51] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24870 and previous config saved to /var/cache/conftool/dbconfig/20220417-130851-ladsgroup.json [13:08:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:11:35] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24871 and previous config saved to /var/cache/conftool/dbconfig/20220417-131135-ladsgroup.json [13:11:37] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [13:11:38] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [13:11:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:11:40] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [13:11:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:11:43] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24872 and previous config saved to /var/cache/conftool/dbconfig/20220417-131143-ladsgroup.json [13:11:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:11:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:22:31] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24873 and previous config saved to /var/cache/conftool/dbconfig/20220417-132230-ladsgroup.json [13:22:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:22:36] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [13:23:57] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24874 and previous config saved to /var/cache/conftool/dbconfig/20220417-132356-ladsgroup.json [13:23:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:28:34] (03PS13) 10Fomafix: Add additional aliases for sr-cyrl and sr-latn next to sr-ec and sr-el [puppet] - 10https://gerrit.wikimedia.org/r/368248 (https://phabricator.wikimedia.org/T117845) [13:28:35] (03PS13) 10Fomafix: Add additional aliases for sr-cyrl and sr-latn next to sr-ec and sr-el [puppet] - 10https://gerrit.wikimedia.org/r/368248 (https://phabricator.wikimedia.org/T117845) [13:37:36] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24875 and previous config saved to /var/cache/conftool/dbconfig/20220417-133736-ladsgroup.json [13:37:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:39:03] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24876 and previous config saved to /var/cache/conftool/dbconfig/20220417-133901-ladsgroup.json [13:39:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:39:08] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance [13:39:10] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance [13:39:11] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance [13:39:11] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [13:39:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:39:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:39:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:39:19] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance [13:39:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:47:54] (NodeTextfileStale) firing: Stale textfile for cloudcontrol2001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [13:48:59] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance [13:49:00] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance [13:49:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:49:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:52:41] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24877 and previous config saved to /var/cache/conftool/dbconfig/20220417-135241-ladsgroup.json [13:52:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:58:21] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance [13:58:22] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1138.eqiad.wmnet with reason: Maintenance [13:58:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:58:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:58:27] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24878 and previous config saved to /var/cache/conftool/dbconfig/20220417-135827-ladsgroup.json [13:58:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:58:31] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [14:07:46] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24879 and previous config saved to /var/cache/conftool/dbconfig/20220417-140746-ladsgroup.json [14:07:48] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance [14:07:49] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance [14:07:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:07:51] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [14:07:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:07:54] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24880 and previous config saved to /var/cache/conftool/dbconfig/20220417-140754-ladsgroup.json [14:07:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:07:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:12:07] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24881 and previous config saved to /var/cache/conftool/dbconfig/20220417-141206-ladsgroup.json [14:12:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:23:06] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance [14:23:08] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1112.eqiad.wmnet with reason: Maintenance [14:23:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:23:09] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [14:23:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:23:12] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [14:23:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:23:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:23:17] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P24882 and previous config saved to /var/cache/conftool/dbconfig/20220417-142316-ladsgroup.json [14:23:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:23:21] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [14:24:28] (03PS1) 10Stang: Increase autoconfirmed threshold to 10 edits on iswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/783445 (https://phabricator.wikimedia.org/T306305) [14:24:29] (03PS1) 10Stang: Increase autoconfirmed threshold to 10 edits on iswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/783445 (https://phabricator.wikimedia.org/T306305) [14:27:12] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24883 and previous config saved to /var/cache/conftool/dbconfig/20220417-142712-ladsgroup.json [14:27:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:32:55] (NodeTextfileStale) firing: (3) Stale textfile for elastic1075:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [14:42:17] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24884 and previous config saved to /var/cache/conftool/dbconfig/20220417-144217-ladsgroup.json [14:42:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:42:24] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P24885 and previous config saved to /var/cache/conftool/dbconfig/20220417-144223-ladsgroup.json [14:42:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:42:28] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [14:57:22] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24886 and previous config saved to /var/cache/conftool/dbconfig/20220417-145722-ladsgroup.json [14:57:24] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance [14:57:26] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance [14:57:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:57:27] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [14:57:27] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [14:57:29] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P24887 and previous config saved to /var/cache/conftool/dbconfig/20220417-145728-ladsgroup.json [14:57:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:57:30] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [14:57:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:57:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:57:35] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24888 and previous config saved to /var/cache/conftool/dbconfig/20220417-145734-ladsgroup.json [14:57:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:57:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:57:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:58:42] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24889 and previous config saved to /var/cache/conftool/dbconfig/20220417-145841-ladsgroup.json [14:58:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:01:55] (NodeTextfileStale) firing: Stale textfile for ms-be2067:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [15:12:34] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P24890 and previous config saved to /var/cache/conftool/dbconfig/20220417-151233-ladsgroup.json [15:12:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:12:53] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24891 and previous config saved to /var/cache/conftool/dbconfig/20220417-151253-ladsgroup.json [15:12:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:12:57] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [15:13:47] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P24892 and previous config saved to /var/cache/conftool/dbconfig/20220417-151346-ladsgroup.json [15:13:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:15:43] (03PS1) 10Gergő Tisza: Add video marketing campaign to $wgGECampaignPattern [mediawiki-config] - 10https://gerrit.wikimedia.org/r/783449 (https://phabricator.wikimedia.org/T303785) [15:15:43] (03PS1) 10Gergő Tisza: Add video marketing campaign to $wgGECampaignPattern [mediawiki-config] - 10https://gerrit.wikimedia.org/r/783449 (https://phabricator.wikimedia.org/T303785) [15:27:39] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298565)', diff saved to https://phabricator.wikimedia.org/P24893 and previous config saved to /var/cache/conftool/dbconfig/20220417-152738-ladsgroup.json [15:27:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:27:44] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [15:27:58] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24894 and previous config saved to /var/cache/conftool/dbconfig/20220417-152758-ladsgroup.json [15:28:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:28:52] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1138', diff saved to https://phabricator.wikimedia.org/P24895 and previous config saved to /var/cache/conftool/dbconfig/20220417-152851-ladsgroup.json [15:28:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:43:03] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24896 and previous config saved to /var/cache/conftool/dbconfig/20220417-154303-ladsgroup.json [15:43:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:43:56] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P24897 and previous config saved to /var/cache/conftool/dbconfig/20220417-154356-ladsgroup.json [15:44:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:44:00] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [15:44:03] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance [15:44:05] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance [15:44:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:44:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:33] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance [15:52:35] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2110.codfw.wmnet with reason: Maintenance [15:52:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:36] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on 12 hosts with reason: Maintenance [15:52:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:52:45] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance [15:52:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:58:08] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24898 and previous config saved to /var/cache/conftool/dbconfig/20220417-155808-ladsgroup.json [15:58:10] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance [15:58:11] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance [15:58:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:58:14] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [15:58:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:58:16] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24899 and previous config saved to /var/cache/conftool/dbconfig/20220417-155816-ladsgroup.json [15:58:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:58:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:01:36] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance [16:01:37] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1121.eqiad.wmnet with reason: Maintenance [16:01:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:01:39] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [16:01:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:01:42] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [16:01:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:01:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:01:47] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24900 and previous config saved to /var/cache/conftool/dbconfig/20220417-160146-ladsgroup.json [16:01:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:13:46] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24901 and previous config saved to /var/cache/conftool/dbconfig/20220417-161346-ladsgroup.json [16:13:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:13:51] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [16:21:08] PROBLEM - BGP status on cr2-eqiad is CRITICAL: BGP CRITICAL - AS64605/IPv4: Active - Anycast https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status [16:28:51] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24902 and previous config saved to /var/cache/conftool/dbconfig/20220417-162851-ladsgroup.json [16:28:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:37:13] (KubernetesRsyslogDown) firing: rsyslog on kubernetes1018:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [16:43:57] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P24903 and previous config saved to /var/cache/conftool/dbconfig/20220417-164356-ladsgroup.json [16:43:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:58:31] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24904 and previous config saved to /var/cache/conftool/dbconfig/20220417-165830-ladsgroup.json [16:58:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:58:35] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [16:59:02] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1121 (T298565)', diff saved to https://phabricator.wikimedia.org/P24905 and previous config saved to /var/cache/conftool/dbconfig/20220417-165901-ladsgroup.json [16:59:03] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance [16:59:05] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance [16:59:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:59:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:59:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:59:10] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24906 and previous config saved to /var/cache/conftool/dbconfig/20220417-165909-ladsgroup.json [16:59:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:09:53] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24907 and previous config saved to /var/cache/conftool/dbconfig/20220417-170952-ladsgroup.json [17:09:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:10:00] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [17:13:36] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24908 and previous config saved to /var/cache/conftool/dbconfig/20220417-171335-ladsgroup.json [17:13:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:24:58] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24909 and previous config saved to /var/cache/conftool/dbconfig/20220417-172457-ladsgroup.json [17:25:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:28:41] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24910 and previous config saved to /var/cache/conftool/dbconfig/20220417-172840-ladsgroup.json [17:28:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:40:03] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P24911 and previous config saved to /var/cache/conftool/dbconfig/20220417-174002-ladsgroup.json [17:40:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:43:46] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1180 (T298565)', diff saved to https://phabricator.wikimedia.org/P24912 and previous config saved to /var/cache/conftool/dbconfig/20220417-174345-ladsgroup.json [17:43:47] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [17:43:49] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance [17:43:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:43:51] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [17:43:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:43:54] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24913 and previous config saved to /var/cache/conftool/dbconfig/20220417-174353-ladsgroup.json [17:43:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:43:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:47:54] (NodeTextfileStale) firing: Stale textfile for cloudcontrol2001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [17:55:08] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1141 (T298565)', diff saved to https://phabricator.wikimedia.org/P24914 and previous config saved to /var/cache/conftool/dbconfig/20220417-175507-ladsgroup.json [17:55:09] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance [17:55:11] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance [17:55:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:55:12] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [17:55:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:55:16] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24915 and previous config saved to /var/cache/conftool/dbconfig/20220417-175515-ladsgroup.json [17:55:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:55:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:06:54] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24916 and previous config saved to /var/cache/conftool/dbconfig/20220417-180653-ladsgroup.json [18:06:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:06:58] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [18:21:59] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24917 and previous config saved to /var/cache/conftool/dbconfig/20220417-182158-ladsgroup.json [18:22:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:32:55] (NodeTextfileStale) firing: (3) Stale textfile for elastic1075:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [18:37:04] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P24918 and previous config saved to /var/cache/conftool/dbconfig/20220417-183703-ladsgroup.json [18:37:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:44:09] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24919 and previous config saved to /var/cache/conftool/dbconfig/20220417-184408-ladsgroup.json [18:44:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:44:13] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [18:52:09] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298565)', diff saved to https://phabricator.wikimedia.org/P24920 and previous config saved to /var/cache/conftool/dbconfig/20220417-185208-ladsgroup.json [18:52:10] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance [18:52:12] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance [18:52:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:52:14] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [18:52:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:52:17] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24921 and previous config saved to /var/cache/conftool/dbconfig/20220417-185216-ladsgroup.json [18:52:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:52:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:59:14] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24922 and previous config saved to /var/cache/conftool/dbconfig/20220417-185913-ladsgroup.json [18:59:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:01:55] (NodeTextfileStale) firing: Stale textfile for ms-be2067:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [19:03:07] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24923 and previous config saved to /var/cache/conftool/dbconfig/20220417-190306-ladsgroup.json [19:03:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:03:11] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [19:14:19] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P24924 and previous config saved to /var/cache/conftool/dbconfig/20220417-191418-ladsgroup.json [19:14:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:18:12] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24925 and previous config saved to /var/cache/conftool/dbconfig/20220417-191811-ladsgroup.json [19:18:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:29:24] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24926 and previous config saved to /var/cache/conftool/dbconfig/20220417-192923-ladsgroup.json [19:29:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:29:29] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [19:29:32] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance [19:29:34] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance [19:29:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:29:35] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [19:29:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:29:38] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [19:29:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:29:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:29:43] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24927 and previous config saved to /var/cache/conftool/dbconfig/20220417-192942-ladsgroup.json [19:29:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:33:17] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P24928 and previous config saved to /var/cache/conftool/dbconfig/20220417-193316-ladsgroup.json [19:33:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:33:56] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24929 and previous config saved to /var/cache/conftool/dbconfig/20220417-193355-ladsgroup.json [19:33:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:48:22] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298565)', diff saved to https://phabricator.wikimedia.org/P24930 and previous config saved to /var/cache/conftool/dbconfig/20220417-194821-ladsgroup.json [19:48:23] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance [19:48:25] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance [19:48:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:48:26] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [19:48:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:48:30] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24931 and previous config saved to /var/cache/conftool/dbconfig/20220417-194829-ladsgroup.json [19:48:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:48:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:49:01] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24932 and previous config saved to /var/cache/conftool/dbconfig/20220417-194900-ladsgroup.json [19:49:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:57:00] PROBLEM - SSH on mw2258.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:59:25] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24933 and previous config saved to /var/cache/conftool/dbconfig/20220417-195924-ladsgroup.json [19:59:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:59:30] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [20:04:06] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P24934 and previous config saved to /var/cache/conftool/dbconfig/20220417-200405-ladsgroup.json [20:04:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:14:30] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24935 and previous config saved to /var/cache/conftool/dbconfig/20220417-201429-ladsgroup.json [20:14:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:19:11] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1165 (T298565)', diff saved to https://phabricator.wikimedia.org/P24936 and previous config saved to /var/cache/conftool/dbconfig/20220417-201910-ladsgroup.json [20:19:12] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance [20:19:14] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance [20:19:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:19:15] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [20:19:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:19:19] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24937 and previous config saved to /var/cache/conftool/dbconfig/20220417-201918-ladsgroup.json [20:19:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:19:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:23:34] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24938 and previous config saved to /var/cache/conftool/dbconfig/20220417-202333-ladsgroup.json [20:23:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:29:35] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P24939 and previous config saved to /var/cache/conftool/dbconfig/20220417-202934-ladsgroup.json [20:29:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:37:13] (KubernetesRsyslogDown) firing: rsyslog on kubernetes1018:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [20:38:39] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24940 and previous config saved to /var/cache/conftool/dbconfig/20220417-203838-ladsgroup.json [20:38:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:44:40] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24941 and previous config saved to /var/cache/conftool/dbconfig/20220417-204439-ladsgroup.json [20:44:41] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance [20:44:43] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance [20:44:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:44:44] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [20:44:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:44:48] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24942 and previous config saved to /var/cache/conftool/dbconfig/20220417-204447-ladsgroup.json [20:44:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:44:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:53:44] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P24943 and previous config saved to /var/cache/conftool/dbconfig/20220417-205343-ladsgroup.json [20:53:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:55:25] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24944 and previous config saved to /var/cache/conftool/dbconfig/20220417-205524-ladsgroup.json [20:55:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:55:29] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [20:58:10] RECOVERY - SSH on mw2258.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:01:16] (03PS1) 10Zabe: scap: remove two absented files [puppet] - 10https://gerrit.wikimedia.org/r/783461 [21:08:49] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1168 (T298565)', diff saved to https://phabricator.wikimedia.org/P24945 and previous config saved to /var/cache/conftool/dbconfig/20220417-210848-ladsgroup.json [21:08:50] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [21:08:52] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1113.eqiad.wmnet with reason: Maintenance [21:08:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:08:54] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [21:08:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:08:57] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24946 and previous config saved to /var/cache/conftool/dbconfig/20220417-210856-ladsgroup.json [21:08:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:09:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:10:30] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24947 and previous config saved to /var/cache/conftool/dbconfig/20220417-211029-ladsgroup.json [21:10:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:20:43] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24948 and previous config saved to /var/cache/conftool/dbconfig/20220417-212042-ladsgroup.json [21:20:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:20:47] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [21:25:35] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P24949 and previous config saved to /var/cache/conftool/dbconfig/20220417-212535-ladsgroup.json [21:25:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:35:48] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24950 and previous config saved to /var/cache/conftool/dbconfig/20220417-213547-ladsgroup.json [21:35:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:40:40] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24951 and previous config saved to /var/cache/conftool/dbconfig/20220417-214040-ladsgroup.json [21:40:42] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [21:40:43] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance [21:40:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:40:45] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [21:40:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:40:48] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24952 and previous config saved to /var/cache/conftool/dbconfig/20220417-214048-ladsgroup.json [21:40:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:40:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:47:54] (NodeTextfileStale) firing: Stale textfile for cloudcontrol2001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [21:50:53] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P24953 and previous config saved to /var/cache/conftool/dbconfig/20220417-215052-ladsgroup.json [21:50:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:55:22] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24954 and previous config saved to /var/cache/conftool/dbconfig/20220417-215521-ladsgroup.json [21:55:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:55:26] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [22:05:58] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1113:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24955 and previous config saved to /var/cache/conftool/dbconfig/20220417-220557-ladsgroup.json [22:05:59] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance [22:06:01] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance [22:06:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:06:02] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [22:06:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:06:06] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24956 and previous config saved to /var/cache/conftool/dbconfig/20220417-220605-ladsgroup.json [22:06:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:06:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:10:27] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24957 and previous config saved to /var/cache/conftool/dbconfig/20220417-221026-ladsgroup.json [22:10:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:18:08] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24958 and previous config saved to /var/cache/conftool/dbconfig/20220417-221808-ladsgroup.json [22:18:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:18:13] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [22:25:32] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P24959 and previous config saved to /var/cache/conftool/dbconfig/20220417-222532-ladsgroup.json [22:25:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:29:54] PROBLEM - Varnish traffic drop between 30min ago and now at esams on alert1001 is CRITICAL: 39.82 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [22:30:10] PROBLEM - Varnish traffic drop between 30min ago and now at eqiad on alert1001 is CRITICAL: 45.02 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [22:31:20] PROBLEM - Varnish traffic drop between 30min ago and now at eqsin on alert1001 is CRITICAL: 32.19 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [22:32:24] RECOVERY - Varnish traffic drop between 30min ago and now at eqiad on alert1001 is OK: (C)60 le (W)70 le 100.8 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [22:32:55] (NodeTextfileStale) firing: (3) Stale textfile for elastic1075:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [22:33:13] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24960 and previous config saved to /var/cache/conftool/dbconfig/20220417-223313-ladsgroup.json [22:33:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:33:36] RECOVERY - Varnish traffic drop between 30min ago and now at eqsin on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [22:34:26] RECOVERY - Varnish traffic drop between 30min ago and now at esams on alert1001 is OK: (C)60 le (W)70 le 82.68 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/d/000000180/varnish-http-requests?orgId=1&viewPanel=6 [22:40:37] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298565)', diff saved to https://phabricator.wikimedia.org/P24961 and previous config saved to /var/cache/conftool/dbconfig/20220417-224037-ladsgroup.json [22:40:39] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [22:40:40] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance [22:40:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:40:42] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [22:40:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:40:45] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24962 and previous config saved to /var/cache/conftool/dbconfig/20220417-224045-ladsgroup.json [22:40:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:40:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:48:18] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P24963 and previous config saved to /var/cache/conftool/dbconfig/20220417-224818-ladsgroup.json [22:48:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:52:24] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24964 and previous config saved to /var/cache/conftool/dbconfig/20220417-225224-ladsgroup.json [22:52:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:52:28] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [22:53:30] Is Quarry down? [23:01:55] (NodeTextfileStale) firing: Stale textfile for ms-be2067:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [23:03:06] not for me [23:03:20] clarification: don't mean down as in offline. rather, down as in queries not running [23:03:23] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T298565)', diff saved to https://phabricator.wikimedia.org/P24965 and previous config saved to /var/cache/conftool/dbconfig/20220417-230323-ladsgroup.json [23:03:25] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [23:03:26] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [23:03:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:03:30] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [23:03:31] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24966 and previous config saved to /var/cache/conftool/dbconfig/20220417-230331-ladsgroup.json [23:03:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:03:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:03:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:03:38] namely, my query and one other have been queued for 18 minutes [23:04:00] ah [23:05:20] https://quarry.wmcloud.org/query/runs/all shows 3 identical queries by Huji, one which ran on dewiki in 8 minutes, the others of which are still listed as running, 2 hours later [23:06:28] PROBLEM - SSH on wtp1045.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [23:07:29] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24967 and previous config saved to /var/cache/conftool/dbconfig/20220417-230729-ladsgroup.json [23:07:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:09:51] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24968 and previous config saved to /var/cache/conftool/dbconfig/20220417-230951-ladsgroup.json [23:09:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:09:55] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [23:22:34] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P24969 and previous config saved to /var/cache/conftool/dbconfig/20220417-232234-ladsgroup.json [23:22:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:24:57] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24970 and previous config saved to /var/cache/conftool/dbconfig/20220417-232456-ladsgroup.json [23:24:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:36:02] PROBLEM - MariaDB Replica IO: s2 on db2101 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2026, Errmsg: error reconnecting to master repl@db2104.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 86400 message: SSL connection error00000000:lib(0):func(0):reason(0) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [23:36:33] hmm. now the query that started right before mine has completed, as have 3 newer ones, but mine is still queued... strange [23:36:46] PROBLEM - MariaDB Replica IO: x1 on db2101 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2026, Errmsg: error reconnecting to master repl@db2096.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 86400 message: SSL connection error00000000:lib(0):func(0):reason(0) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [23:36:46] PROBLEM - MariaDB Replica IO: s5 on db2101 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2026, Errmsg: error reconnecting to master repl@db2123.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 86400 message: SSL connection error00000000:lib(0):func(0):reason(0) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [23:37:40] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298565)', diff saved to https://phabricator.wikimedia.org/P24971 and previous config saved to /var/cache/conftool/dbconfig/20220417-233739-ladsgroup.json [23:37:41] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance [23:37:43] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance [23:37:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:37:46] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [23:37:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:37:49] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Depooling db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24972 and previous config saved to /var/cache/conftool/dbconfig/20220417-233747-ladsgroup.json [23:37:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:37:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:38:20] RECOVERY - MariaDB Replica IO: s2 on db2101 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [23:39:04] RECOVERY - MariaDB Replica IO: x1 on db2101 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [23:39:04] RECOVERY - MariaDB Replica IO: s5 on db2101 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [23:40:02] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P24973 and previous config saved to /var/cache/conftool/dbconfig/20220417-234001-ladsgroup.json [23:40:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:48:56] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T298565)', diff saved to https://phabricator.wikimedia.org/P24974 and previous config saved to /var/cache/conftool/dbconfig/20220417-234856-ladsgroup.json [23:48:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:49:01] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [23:55:07] !log ladsgroup@cumin1001 dbctl commit (dc=all): 'Repooling after maintenance db1131 (T298565)', diff saved to https://phabricator.wikimedia.org/P24975 and previous config saved to /var/cache/conftool/dbconfig/20220417-235506-ladsgroup.json [23:55:08] !log ladsgroup@cumin1001 START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [23:55:10] !log ladsgroup@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1005.eqiad.wmnet with reason: Maintenance [23:55:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:55:11] T298565: Fix mismatching field type of user table for columns user_email_authenticated, user_email_token, user_email_token_expires, user_newpass_time, user_registration, user_token, user_touched, user_newpassword, user_password, user_email on wmf wikis - https://phabricator.wikimedia.org/T298565 [23:55:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:55:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log