[00:31:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: CI needs to update some test with @covers - https://phabricator.wikimedia.org/T384656#10490950 (10AnnWF) [00:37:33] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for ebrill - https://phabricator.wikimedia.org/T384665 (10Dwisehaupt) 03NEW [00:37:35] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for sborriello - https://phabricator.wikimedia.org/T384666 (10Dwisehaupt) 03NEW [00:37:37] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for areyes - https://phabricator.wikimedia.org/T384667 (10Dwisehaupt) 03NEW [00:37:38] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for ppenloglou - https://phabricator.wikimedia.org/T384668 (10Dwisehaupt) 03NEW [00:37:40] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for lwadleigh - https://phabricator.wikimedia.org/T384669 (10Dwisehaupt) 03NEW [00:37:41] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for dbautista - https://phabricator.wikimedia.org/T384670 (10Dwisehaupt) 03NEW [00:37:43] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for ejegg - https://phabricator.wikimedia.org/T384671 (10Dwisehaupt) 03NEW [00:37:44] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for jschoenig - https://phabricator.wikimedia.org/T384672 (10Dwisehaupt) 03NEW [00:50:44] (03CR) 10Ladsgroup: [C:03+2] Use ContentLanguageCode service instead of heavy Language object [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1094549 (https://phabricator.wikimedia.org/T376565) (owner: 10Jforrester) [00:57:30] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: CI needs to update some test for php 8.2 upgrade - https://phabricator.wikimedia.org/T384656#10491079 (10AnnWF) [01:09:12] (03Merged) 10jenkins-bot: Use ContentLanguageCode service instead of heavy Language object [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1094549 (https://phabricator.wikimedia.org/T376565) (owner: 10Jforrester) [01:25:47] 10fundraising-tech-ops: Add new ssh key for greg - https://phabricator.wikimedia.org/T384673 (10greg) 03NEW p:05Triage→03Medium [14:15:16] (03PS3) 10Damilare Adedoyin: Update ThankYou tests to use APIv4 send, to complete the phasing off of the thank_you_for_contribution function in the thank_you.module [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1113156 (https://phabricator.wikimedia.org/T270674) [14:17:57] (03CR) 10Damilare Adedoyin: "Thanks jgleeson for the review. I've updated the test and added some context in the reply." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1113156 (https://phabricator.wikimedia.org/T270674) (owner: 10Damilare Adedoyin) [14:22:14] thanks damilare will check it out [14:36:02] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install fransc1001 - https://phabricator.wikimedia.org/T367814#10492369 (10cmooney) @VRiley-WMF ok so after a bit more back-and-forth I think we can finally trial this new script and see how it works: https://netbox-next.wikimedia.org/ext... [15:30:00] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install fransc1001 - https://phabricator.wikimedia.org/T367814#10492525 (10cmooney) [15:32:18] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install fransc1001 - https://phabricator.wikimedia.org/T367814#10492529 (10cmooney) [15:32:47] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install fransc1001 - https://phabricator.wikimedia.org/T367814#10492531 (10cmooney) [15:34:22] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install fransc1001 - https://phabricator.wikimedia.org/T367814#10492540 (10cmooney) [17:19:10] 06Fundraising-Backlog: Recurring queue consumer PHP notice clean up - auto rescue failure - https://phabricator.wikimedia.org/T384729 (10Cstone) 03NEW [17:19:32] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Recurring queue consumer PHP notice clean up - auto rescue failure - https://phabricator.wikimedia.org/T384729#10493090 (10Cstone) a:03Cstone [17:21:27] 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for spatton - https://phabricator.wikimedia.org/T384354#10493100 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms. [18:39:25] https://gerrit.wikimedia.org/r/c/integration/config/+/1113583/1 [18:39:28] argh [18:39:43] fr-tech, wfan: it looks like it's not as simple as reverting one patch to bring back our old CI. the revert patch is failing due to the old job builds no longer existing :| they were removed here https://gerrit.wikimedia.org/r/c/integration/config/+/1113584/2 and I also realised there's a bigger patch chain which was added to upgrade our build to 8.2 that might not be as easy to roll back! [18:39:43] https://gerrit.wikimedia.org/r/c/integration/config/+/1113583/1 [18:40:25] hmmm [18:40:46] maybe we do need to just suck it up and make those upgrades you added in the ticket wfan [18:47:05] At least I know the cover one. Ok thanks for looking into that Jack! [18:50:16] wfan: it might be easier to split those upgrade tasks out into a separate precursor patch and then rebase your chain over them [18:58:59] 10fundraising-tech-ops: Add new ssh key for greg - https://phabricator.wikimedia.org/T384673#10493421 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt key added to puppet and merged. should be on the hosts with their next puppet runs. [19:04:56] sure~ so we had to solve this first then those epc patches [19:27:27] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: CI needs to update some test for php 8.2 upgrade - https://phabricator.wikimedia.org/T384656#10493535 (10AnnWF) [19:35:42] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: CI needs to update some test for php 8.2 upgrade - https://phabricator.wikimedia.org/T384656#10493609 (10AnnWF) [19:36:56] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: CI needs to update some test for php 8.2 upgrade - https://phabricator.wikimedia.org/T384656#10493613 (10AnnWF) [19:45:32] (03PS1) 10Wfan: Do not use null as preg_match second parm [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1114040 (https://phabricator.wikimedia.org/T384656) [19:46:27] (03PS2) 10Wfan: Do not use null as preg_match second param [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1114040 (https://phabricator.wikimedia.org/T384656) [19:47:05] (03PS3) 10Wfan: Passing non-nullable utm_source as preg_match second param [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1114040 (https://phabricator.wikimedia.org/T384656) [19:48:46] (03CR) 10CI reject: [V:04-1] Passing non-nullable utm_source as preg_match second param [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1114040 (https://phabricator.wikimedia.org/T384656) (owner: 10Wfan) [19:59:56] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Recurring queue consumer PHP notice clean up - auto rescue failure - https://phabricator.wikimedia.org/T384729#10493690 (10Cstone) [20:09:32] 06Fundraising-Backlog: Automatic merging of organization records with same name - https://phabricator.wikimedia.org/T384748 (10MDemosWMF) 03NEW [20:15:12] PROBLEM - check_memory on fran2001 is CRITICAL: CRIT Memory 99% used. Largest process: python3 (949209) = 88.1% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_memory [20:20:10] PROBLEM - check_memory on fran2001 is CRITICAL: CRIT Memory 98% used. Largest process: python3 (949209) = 87.7% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_memory [20:22:37] (03PS1) 10Umherirrender: Replace call_user_func_array with dynamic function call [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1114043 [20:25:10] PROBLEM - check_memory on fran2001 is CRITICAL: CRIT Memory 99% used. Largest process: python3 (949209) = 87.7% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_memory [20:26:48] going to ack that, it's a jupyter notebook chewing up memory while it runs. [20:27:45] ACKNOWLEDGEMENT - check_memory on fran2001 is CRITICAL: CRIT Memory 99% used. Largest process: python3 (949209) = 87.7% Dwisehaupt known - jupyter notebook grooving along. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_memory [20:38:21] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Recurring queue consumer PHP notice clean up - auto rescue failure - https://phabricator.wikimedia.org/T384729#10493764 (10AnnWF) related with https://phabricator.wikimedia.org/T382833 and would be fixed by the deployment of https://gerrit.wikimedia.org/r... [21:03:34] (03PS1) 10Umherirrender: Replace call_user_func_array with dynamic function call [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1114059 [21:05:35] (03CR) 10CI reject: [V:04-1] Replace call_user_func_array with dynamic function call [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1114059 (owner: 10Umherirrender) [21:29:57] (03CR) 10Umherirrender: Passing non-nullable utm_source as preg_match second param (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1114040 (https://phabricator.wikimedia.org/T384656) (owner: 10Wfan) [21:56:16] PROBLEM - check_load on fran2001 is CRITICAL: LOAD CRITICAL - total load average: 66.82, 39.92, 18.70 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load [22:02:28] PROBLEM - check_load on fran2001 is CRITICAL: LOAD CRITICAL - total load average: 59.41, 45.26, 26.35 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load [22:07:10] PROBLEM - check_load on fran2001 is CRITICAL: LOAD CRITICAL - total load average: 61.94, 49.65, 33.78 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load [22:10:10] PROBLEM - check_load on fran2001 is CRITICAL: LOAD CRITICAL - total load average: 23.65, 41.73, 33.76 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load [22:15:08] PROBLEM - check_load on fran2001 is CRITICAL: LOAD CRITICAL - total load average: 0.35, 15.44, 24.50 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load [22:15:08] RECOVERY - check_memory on fran2001 is OK: OK Memory 8% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_memory [22:45:08] RECOVERY - check_load on fran2001 is OK: LOAD OK - total load average: 0.15, 0.25, 3.71 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_load