[00:08:49] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10294866 (10Legoktm) >>! In T374683#10294780, @Legoktm wrot... [09:34:10] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10295353 (10daniel) Bill correctly identified the root caus... [09:59:42] legoktm: ouch. sorry about that. I wasn't aware this hadn't been communicated further, I assume it was because it was thought it would be a 0 impact. In retrospect, it should have. [10:15:22] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10295502 (10Joe) >>! In T374683#10295353, @daniel wrote: >... [10:35:42] 06serviceops, 10Maps (Kartotherian), 13Patch-For-Review: Strategy to slowly move Kartotherian's traffic from bare metal to k8s - https://phabricator.wikimedia.org/T378944#10295540 (10elukey) I had a chat with @Jgiannelos about the plan and it seems good, we just need to verify if the old plaintext lvs:ip com... [10:54:45] 06serviceops, 06Content-Transform-Team, 10WMDE-TechWish-Maintenance, 07Epic, and 2 others: Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#10295605 (10elukey) To keep archives happy, I met with @Jgiannelos and we decided to proceed in this way: * We keep working on having a Doc... [11:04:52] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10295634 (10daniel) >>! In T374683#10295502, @Joe wrote: >... [13:18:53] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10295965 (10CDanis) >>! In T374683#10295634, @daniel wrote:... [16:23:04] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10296723 (10fnegri) A similar issue was reported today by @Ederporto in [[ https://wm-bot.wmcloud.org/logs/%23wikimedia-cloud/2... [16:32:20] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10296766 (10TheDJ) Errors are clearly spiking again https://grafana.wikimedia.org/d/U4TuF-lMk/proton?orgId=1&from=173030579182... [16:32:39] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10296770 (10TheDJ) p:05Triage→03Unbreak! [16:33:26] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10296778 (10hnowlan) Looks like the same crashpad flood issue again. The service needs a restart, and I think we should impleme... [16:35:13] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10296784 (10Legoktm) >>! In T374683#10295353, @daniel wrote... [17:27:55] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10297015 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host mc-gp2006.codfw.wmnet with OS bookworm [19:57:29] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10297564 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host mc-gp2006.codfw.wmnet with OS bookworm completed: - mc-gp2006 (**WARN**... [19:57:44] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10297568 (10Jhancock.wm) [19:58:46] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10297569 (10Jhancock.wm) 05Open→03Resolved @Clement_Goubert this is ready for y'all [20:02:20] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297575 (10Jhancock.wm) [20:33:30] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10297611 (10Xaosflux) [20:40:24] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297624 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2146.codfw.wmnet with O... [20:40:26] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297625 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2147.codfw.wmnet with O... [20:40:30] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297626 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2149.codfw.wmnet with O... [20:40:31] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297627 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2150.codfw.wmnet with O... [20:40:36] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297628 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2148.codfw.wmnet with O... [20:46:11] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10297633 (10Quiddity) It seems like this should be included... [21:21:08] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10297760 (10BPirkle) >> Also, we may encounter problems tha... [21:21:46] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10297761 (10daniel) > The responses were also not returning... [21:26:59] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297771 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2149.codfw.wmnet with OS bo... [21:27:00] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297772 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2146.codfw.wmnet with OS bo... [21:27:40] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297774 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2147.codfw.wmnet with OS bo... [21:31:50] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297789 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2148.codfw.wmnet with OS bo... [21:43:13] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297808 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2150.codfw.wmnet with OS bo... [22:11:13] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297878 (10Jhancock.wm) [22:25:15] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032#10297909 (10Jclark-ctr) [22:34:11] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review, 07Wikimedia-Incident: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10297946 (10Legoktm) >>! In T374683#10297761, @daniel wrote... [22:39:52] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297954 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2151.codfw.wmnet with O... [22:39:57] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297955 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2152.codfw.wmnet with O... [22:39:59] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297956 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2153.codfw.wmnet with O... [22:40:01] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297957 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2154.codfw.wmnet with O... [22:40:03] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10297958 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2155.codfw.wmnet with O... [22:44:00] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032#10297962 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host mc-gp1006.eqiad.wmnet with OS bookworm [22:44:08] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032#10297964 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host mc-gp1005.eqiad.wmnet with OS bookworm [22:44:21] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032#10297965 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host mc-gp1004.eqiad.wmnet with OS bookworm [23:23:31] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10298075 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2155.codfw.wmnet with OS bo... [23:24:10] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032#10298076 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host mc-gp1004.eqiad.wmnet with OS bookworm completed: - mc-gp1004 (**PASS**) -... [23:29:10] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10298086 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2153.codfw.wmnet with OS bo... [23:31:39] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032#10298090 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host mc-gp1005.eqiad.wmnet with OS bookworm completed: - mc-gp1005 (**PASS**) -... [23:36:14] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10298098 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2154.codfw.wmnet with OS bo... [23:39:58] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10298108 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2151.codfw.wmnet with OS bo... [23:41:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032#10298120 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host mc-gp1006.eqiad.wmnet with OS bookworm completed: - mc-gp1006 (**PASS**) -... [23:46:25] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10298135 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2152.codfw.wmnet with OS bo... [23:46:39] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10298136 (10Jhancock.wm)