[00:36:09] FIRING: [8x] LVSHighCPU: The host lvs5005:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [00:41:09] RESOLVED: [8x] LVSHighCPU: The host lvs5005:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [01:25:38] greg-g: I have responded on the task on the Traffic side of things and the hesitancy to do subdomains of w.wiki even. please let me know if there are any questions. [01:25:59] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 07fr-acoustic: Acoustic SMS: Domain needed for short links - https://phabricator.wikimedia.org/T379318#10533361 (10ssingh) [02:13:04] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10533385 (10Scott_French) Thank you both so much for your debugging today, @Daimona and @mszabo! Great, so it sounds like we have a sol... [03:04:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [03:09:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [04:09:51] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10533418 (10Daimona) >>! In T385395#10533126, @mszabo wrote: > @daniel was right—this is indeed a case of catastrophic backtracking, al... [04:35:15] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10533423 (10mszabo) >>! In T385395#10533418, @Daimona wrote: >>>! In T385395#10533126, @mszabo wrote: >> @daniel was right—this is inde... [13:35:09] FIRING: [8x] LVSHighCPU: The host lvs7002:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs7002 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [13:40:09] RESOLVED: [8x] LVSHighCPU: The host lvs7002:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs7002 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [14:13:16] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10533617 (10mszabo) The PCRE version used by our PHP 7.4 base image is 10.32, while 8.1 uses 10.36: ` $ podman run --rm docker-registry... [14:16:02] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade network devices to Junos 20+ - https://phabricator.wikimedia.org/T316539#10533621 (10taavi) Anything left to do here? [15:43:14] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10533683 (10Daimona) >>! In T385395#10533423, @mszabo wrote: >>>! In T385395#10533418, @Daimona wrote: >>>>! In T385395#10533126, @msza... [16:15:39] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10533696 (10mszabo) >>! In T385395#10533683, @Daimona wrote: > Yeah, fair point. RE2 was even considered as an option in T240884 5 yea... [16:23:14] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10533700 (10mszabo) As for a potential way forward: It seems PHP 8.1 bundles PCRE2 10.39, which includes the above fix, so we could bui... [20:15:47] 06Traffic, 06MediaWiki-Engineering, 06serviceops, 07Upstream, 07Wikimedia-production-error: 503 error when edit large size pages on PHP 8.1 - https://phabricator.wikimedia.org/T385395#10533797 (10Reedy) [20:44:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [20:49:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX