[00:11:22] 10fundraising-tech-ops: Fundraising access request for cmooney - https://phabricator.wikimedia.org/T374561#10232037 (10Dwisehaupt) [00:12:17] 10fundraising-tech-ops: Fundraising access request for cmooney - https://phabricator.wikimedia.org/T374561#10232038 (10Dwisehaupt) @cmooney Thanks. I have added your ssh and yubikey pub keys to our config and pushed the changes out. You should not be able to test your ssh connections. I'll send an email with the... [00:19:36] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10232047 (10Dwisehaupt) [00:21:28] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10232049 (10Dwisehaupt) Access request approval: ` Date: Fri, 11 Oct 2024 14:02:59 From: Lisa Seitz Gruwell To: Erica Roden Cc: Dallas Wisehaupt , Greg Grossmeier , Jeff Green Subject: Re: Approval system ac... [00:24:27] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10232064 (10Dwisehaupt) @ERoden-WMF Do you have an idea yet which databases Ilse will need access to? This will require the use of ssh if direct DB access is required. They were previously granted some acces... [01:12:38] (03CR) 10Ejegg: [C:03+2] Export preference_tags as part of nightly update [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1080097 (https://phabricator.wikimedia.org/T366212) (owner: 10Eileen) [01:13:00] thanks ejegg|away [01:14:18] (03Merged) 10jenkins-bot: Export preference_tags as part of nightly update [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1080097 (https://phabricator.wikimedia.org/T366212) (owner: 10Eileen) [01:32:49] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1080423 [01:33:09] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1080423 (owner: 10Eileen) [01:34:11] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/tools into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1080423 (owner: 10Eileen) [01:50:24] !log tools upgraded from 62f2d170 to 68f64e43 [01:50:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:37:22] 03Fundraising Sprint: trickOrTreat(), 06Fundraising-Backlog: Acoustic SMS stage 4 Acoustic immediately post-convert Look at renaming fields in Acoustic export csv - https://phabricator.wikimedia.org/T376360#10232359 (10Eileenmcnaughton) [03:37:29] 03Fundraising Sprint: trickOrTreat(), 06Fundraising-Backlog: Acoustic SMS stage 4 Acoustic immediately post-convert Look at renaming fields in Acoustic export csv - https://phabricator.wikimedia.org/T376360#10232360 (10Eileenmcnaughton) 05Open→03Resolved [04:34:20] (03PS1) 10Eileen: php8.x deprecation fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1080426 [07:45:22] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06Content-Transform-Team-WIP, 06Language and Product Localization, and 5 others: JavaScript building message boxes should now use mw.util.messageBox - https://phabricator.wikimedia.org/T375902#10232544 (10Nikerabbit) [12:41:49] (03PS1) 10Damilare Adedoyin: Adjust dlocal validation ranges till we get feedback from Dlocal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080661 (https://phabricator.wikimedia.org/T370583) [12:42:45] jgleeson: if you'd be deploying DI too, please help review this to adjust the dlocal validations. [12:43:06] (03CR) 10CI reject: [V:04-1] Adjust dlocal validation ranges till we get feedback from Dlocal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080661 (https://phabricator.wikimedia.org/T370583) (owner: 10Damilare Adedoyin) [12:45:29] (03CR) 10Damilare Adedoyin: "Recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080661 (https://phabricator.wikimedia.org/T370583) (owner: 10Damilare Adedoyin) [12:48:48] (03PS1) 10Jgleeson: Update SmashPig to v0.8.15 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080684 [12:49:58] (03CR) 10CI reject: [V:04-1] Update SmashPig to v0.8.15 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080684 (owner: 10Jgleeson) [12:56:33] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10233385 (10ERoden-WMF) thanks @Dwisehaupt. Previous access was granted when she was a consultant, she will need this access linked to her Wikimedia credentials. @greg per our conversation last week could you... [12:56:38] hmm CI failures dami related to parallel phpunit [12:56:41] damilare: * [13:00:23] asking for help in releng [13:01:13] ah I see this got merged damilare https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1073270 [13:01:22] and we did have to reduce it down in your follow on [13:01:37] always felt wonky expecting 6 to me [13:02:01] I can +2 https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1080661 once we figure out the CI issues [13:15:09] /nick ejegg [13:16:37] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: codfw:frack pfw3 and old fasw decommission - https://phabricator.wikimedia.org/T377254#10233448 (10Papaul) [13:20:33] fr-tech CI on DI is broke [13:20:38] speaking to releng about it [13:20:46] looks like to be ironically related to parallel phpunuit [13:25:33] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: CI failing for DonationInterface - https://phabricator.wikimedia.org/T377327 (10jgleeson) 03NEW [13:26:27] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: CI failing for DonationInterface - https://phabricator.wikimedia.org/T377327#10233476 (10jgleeson) a:03jgleeson [13:32:22] yep thanks jgleeson [13:35:45] TIL https://en.wikipedia.org/wiki/IP_over_Avian_Carriers [13:44:00] (03PS1) 10Zoranzoki21: PHPCS: Fix all excluded rules [extensions/LandingCheck] - 10https://gerrit.wikimedia.org/r/1080712 [14:03:26] (03PS1) 10Ejegg: Use mw.util.messageBox [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1080720 (https://phabricator.wikimedia.org/T375902) [14:04:33] ok fr-tech ^^^ should get CentralNotice in line with that JS / HTML change that was requested [14:04:45] oops, forgot to take the flag down [14:06:21] (03CR) 10CI reject: [V:04-1] Use mw.util.messageBox [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1080720 (https://phabricator.wikimedia.org/T375902) (owner: 10Ejegg) [14:08:28] (03CR) 10Hashar: "recheck after having deployed https://gerrit.wikimedia.org/r/c/integration/config/+/1080450 for T377327" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080684 (owner: 10Jgleeson) [14:15:34] (03PS2) 10Ejegg: Use mw.util.messageBox [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1080720 (https://phabricator.wikimedia.org/T375902) [14:28:05] (03CR) 10CI reject: [V:04-1] Use mw.util.messageBox [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1080720 (https://phabricator.wikimedia.org/T375902) (owner: 10Ejegg) [14:31:16] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: CI failing for DonationInterface - https://phabricator.wikimedia.org/T377327#10233799 (10hashar) 05Open→03Resolved CI no more tries to run PHPUnit in parallel and after having deployed the CI config change, it now passes on https://gerrit.wikimedi... [14:34:36] 03Fundraising Sprint: trickOrTreat(): Ensure Gravy one time tests only initiate one time donations - https://phabricator.wikimedia.org/T377335 (10Damilare) 03NEW [14:34:41] 03Fundraising Sprint: trickOrTreat(), 06Fundraising-Backlog: Ensure Gravy one time tests only initiate one time donations - https://phabricator.wikimedia.org/T377335#10233845 (10Damilare) [14:35:02] (03PS3) 10Ejegg: Use mw.util.messageBox [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1080720 (https://phabricator.wikimedia.org/T375902) [14:35:35] ok, that should satisfy the linter ^^^ [14:41:01] (03PS1) 10Damilare Adedoyin: Ensure no recurring is initiated for gravy one-time only methods. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080730 (https://phabricator.wikimedia.org/T377335) [14:43:06] (03CR) 10CI reject: [V:04-1] Ensure no recurring is initiated for gravy one-time only methods. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080730 (https://phabricator.wikimedia.org/T377335) (owner: 10Damilare Adedoyin) [15:05:15] 10fundraising-tech-ops: Fundraising access request for cmooney - https://phabricator.wikimedia.org/T374561#10234020 (10cmooney) Thanks @Dwisehaupt ! I can confirm all seems to be working fine thanks, I can ssh on and get a stern warning from a scary-looking cow. I guess we can close this task or is there anyth... [15:36:06] CI is now fixed fr-tech [15:36:17] for DonationInterface [15:50:57] (03CR) 10Damilare Adedoyin: "Recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080730 (https://phabricator.wikimedia.org/T377335) (owner: 10Damilare Adedoyin) [15:53:04] (03PS1) 10Damilare Adedoyin: Set error page for email preference center [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080751 (https://phabricator.wikimedia.org/T375798) [15:55:55] (03CR) 10CI reject: [V:04-1] Set error page for email preference center [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080751 (https://phabricator.wikimedia.org/T375798) (owner: 10Damilare Adedoyin) [16:00:52] (03PS2) 10Damilare Adedoyin: Set error page for email preference center [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080751 (https://phabricator.wikimedia.org/T375798) [16:15:28] (03CR) 10Damilare Adedoyin: "Recheck" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080661 (https://phabricator.wikimedia.org/T370583) (owner: 10Damilare Adedoyin) [17:25:43] 10fundraising-tech-ops, 03BDC-Implementation, 06Privacy Engineering, 10Security Preview, 07SecTeam-Processed: Investigate Superset packaging, install, security monitoring. - https://phabricator.wikimedia.org/T368108#10234966 (10Irisbdc) [17:30:17] (03CR) 10Jgleeson: [C:03+2] "LGTM!" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080661 (https://phabricator.wikimedia.org/T370583) (owner: 10Damilare Adedoyin) [17:30:34] I'll add that to the latest deploy damilare [17:30:46] TY! [17:30:52] np [17:31:51] 06Fundraising-Backlog: [low priority] Investigate conditional Google's script addition for Google Ads Grant (fundraising) - https://phabricator.wikimedia.org/T374079#10235022 (10MSuijkerbuijk_WMF) @AKanji-WMF Google Ads run mainly in the U.S., where most of the volume comes from. We also have UK, CA and AU but n... [17:37:02] (03Merged) 10jenkins-bot: Adjust dlocal validation ranges till we get feedback from Dlocal [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080661 (https://phabricator.wikimedia.org/T370583) (owner: 10Damilare Adedoyin) [17:52:52] 06Fundraising-Backlog, 07payments-orchestration: Map Authorization Successful for ACH to complete - https://phabricator.wikimedia.org/T377376 (10Damilare) 03NEW [17:52:56] 06Fundraising-Backlog, 07payments-orchestration: Map Authorization Successful for ACH to complete - https://phabricator.wikimedia.org/T377376#10235118 (10Damilare) [18:10:15] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences: Add unit tests for our preferences centre code using check-sum-authenticated, low permission user - https://phabricator.wikimedia.org/T352926#10235215 (10AKanji-WMF) bumped for higher-priority preference work. [18:10:53] 06Fundraising-Backlog: Coding changes to offline gifts - https://phabricator.wikimedia.org/T372467#10235232 (10AKanji-WMF) @NNichols would you be able to join CiviFortnightly to discuss this phab with us? [18:14:12] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Recurring-Donations: Improvements to failed recurring charge logic - https://phabricator.wikimedia.org/T356073#10235259 (10AKanji-WMF) Targeting to work on this in Sprint U [18:15:53] fr-tech here's a settings change to make wikitech its own campaign target in CentralNotice: https://gerrit.wikimedia.org/r/1080780 [18:16:10] 06Fundraising-Backlog: Update import code (inc upstream) to permit importing last 4 digits of CC number (to financial_trxn.pan_truncation) - https://phabricator.wikimedia.org/T373061#10235269 (10AKanji-WMF) Targeting Sprint U in anticipation of DM returns October/November. [18:16:22] if anyone wants to give it a C+1 I can try to get it deployed (settings repo changes don't get a C+2 till the deploy window) [18:17:17] got it ejegg [18:17:18] 06Fundraising-Backlog: Make gatewaychooser result page for no valid combo found friendlier - https://phabricator.wikimedia.org/T376767#10235274 (10AKanji-WMF) Moving to next based on current priority mix for Sprint U [18:18:47] 06Fundraising-Backlog: Add the chargeback reason from Adyen chargeback IPNs - https://phabricator.wikimedia.org/T353415#10235279 (10AKanji-WMF) Would be good to do this cal year, however bumping from Sprint U based on other priorities. [18:18:53] thanks cstone [18:20:12] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Get stats on number of donations resolved via Pending Transaction Resolver - https://phabricator.wikimedia.org/T374395#10235287 (10AKanji-WMF) Hi @Ejegg - could you please share more context in this phab? tx [18:24:38] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Get stats on number of donations resolved via Pending Transaction Resolver - https://phabricator.wikimedia.org/T374395#10235303 (10Ejegg) It would be nice to write a prometheus file at the end of each run with a count of resolved pending transactions per... [18:27:14] 06Fundraising-Backlog: Search kit fields for pledges - https://phabricator.wikimedia.org/T361635#10235307 (10AKanji-WMF) Targeting Sprint V [18:27:30] fr-tech I've added the pending transaction resolver job for gravy on frpm1002. Could someone take a peek before I deploy it pls? [18:27:55] taking a look jgleeson [18:28:10] ty [18:29:05] oh right, we have a separate job for each one [18:29:36] looks good to me jgleeson ! [18:29:43] ty! [18:29:57] thanks for the C+1 cstone ! Got it scheduled for the deploy window in 1.5 hrs [18:56:32] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: eqiad:frack network design, installation and configuration - https://phabricator.wikimedia.org/T377381 (10cmooney) 03NEW p:05Triage→03Medium [18:56:45] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: eqiad:frack network design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235421 (10cmooney) [18:56:46] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Q1:eqiad:frack network upgrade tracking task - https://phabricator.wikimedia.org/T371435#10235422 (10cmooney) [19:21:15] (03CR) 10Jgleeson: [C:03+2] Update SmashPig to v0.8.15 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080684 (owner: 10Jgleeson) [19:23:02] (03Merged) 10jenkins-bot: Update SmashPig to v0.8.15 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080684 (owner: 10Jgleeson) [19:37:17] (03PS1) 10Jgleeson: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1080800 [19:49:38] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235538 (10cmooney) [19:54:23] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235556 (10cmooney) @jclark-ctr I think existing stock was used for the 100G links between the switches in... [19:57:26] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235573 (10cmooney) [20:02:44] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10235582 (10cmooney) [20:14:34] (03CR) 10Jgleeson: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1080800 (owner: 10Jgleeson) [20:28:47] (03PS1) 10Jgleeson: Update DI, vendor and composer.lock to pull in SmashPig v0.8.15 [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/1080809 [20:45:16] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1080800 (owner: 10Jgleeson) [20:45:54] (03CR) 10Jgleeson: [C:03+2] Update DI, vendor and composer.lock to pull in SmashPig v0.8.15 [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/1080809 (owner: 10Jgleeson) [20:46:05] (03PS3) 10Damilare Adedoyin: Ensure no recurring is initiated for gravy one-time only methods. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1080730 (https://phabricator.wikimedia.org/T377335) [20:49:14] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10wikitech.wikimedia.org: Wikitech showing Wikipedia CentralNotice banners - https://phabricator.wikimedia.org/T377030#10235804 (10Ejegg) Hi @Pcoombe, I just got a settings patch deployed to add wikitech as a sepa... [20:54:53] (03Merged) 10jenkins-bot: Update DI, vendor and composer.lock to pull in SmashPig v0.8.15 [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/1080809 (owner: 10Jgleeson) [20:55:16] does everyone get this error on their local civi status screen? [20:55:17] "WMF CiviCRM" (wmf-civicrm) has a missing dependency on "Recent Items Menu" (org.civicrm.recentmenu) [20:58:07] i think that was part of eileens cleanup I +2d [20:58:24] you might have to install it? [21:00:47] ah thanks [21:08:32] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10235888 (10greg) My initial reaction is: "what Joseph has". [21:57:49] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10236007 (10cmooney) We should also use SFP28 modules for the 'fabric-link' between the two pfw's if possib... [22:11:22] 10fundraising-tech-ops: Fundraising Access For Ilse Ackerman - https://phabricator.wikimedia.org/T377039#10236029 (10JMando) @Dwisehaupt Same access as me should be correct (civicrm, drupal, all the analytics and analytics dev schemas). And same permissions as me as well (so read and write for the analytics sche... [22:27:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [22:32:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [22:37:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [22:42:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [22:47:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [22:52:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [22:57:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:02:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:07:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:12:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:12:30] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10236211 (10cmooney) @RobH so looking at the options after discussion I think we need to do an fs.com (or a... [23:17:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:22:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:27:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:32:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:37:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:42:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:47:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:52:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [23:57:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql