[00:11:08] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:43:55] 06Labs, 10Labs-Infrastructure, 10DBA: Data integrity issue with enwiki_p user_groups on Wikimedia Tool Labs (missing rows) - https://phabricator.wikimedia.org/T159493#3072494 (10MZMcBride) Thanks for the quick fix! I replied to the two users who were asking about the issue: RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [01:32:39] 06Labs: Close wmve-techteam - https://phabricator.wikimedia.org/T159589#3072574 (10White-Master) [02:07:07] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:42:05] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [03:41:25] 10Striker, 10Tool-Labs-tools-stewardbots, 15User-bd808: Link existing repo in Toolsadmin - https://phabricator.wikimedia.org/T159344#3072706 (10bd808) 05Open>03Resolved a:03bd808 https://toolsadmin.wikimedia.org/tools/id/stewardbots/repos/id/labs-tools-stewardbots [03:43:05] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [03:52:21] 10Striker, 10Tool-Labs-tools-Quentinv57's-tools, 15User-bd808: Fix repo in toolsadmin after being renamed in Phab - https://phabricator.wikimedia.org/T159343#3072712 (10bd808) 05Open>03Resolved a:03bd808 https://toolsadmin.wikimedia.org/tools/id/quentinv57-tools/repos/id/labs-tools-quentinv57-tools #S... [03:53:34] 10Striker, 10Tool-Labs-tools-stewardbots, 15User-bd808: Link existing repo in Toolsadmin - https://phabricator.wikimedia.org/T159344#3072721 (10bd808) [03:53:36] 10Striker, 07Epic, 07Tracking: Manually connect existing Diffusion repos to tools in toolsadmin - https://phabricator.wikimedia.org/T144282#3072720 (10bd808) [04:43:07] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [06:09:04] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:33:18] PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:44:06] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [07:08:17] RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [07:40:05] 06Labs, 10Tool-Labs, 06Developer-Relations: Make sure abandoned useful tools are properly advertised so potentially interested new maintainers could find them - https://phabricator.wikimedia.org/T159595#3072793 (10zhuyifei1999) [07:40:26] 06Labs, 10Tool-Labs, 06Developer-Relations: Make sure abandoned useful tools are properly advertised so potentially interested new maintainers could find them - https://phabricator.wikimedia.org/T159595#3072793 (10zhuyifei1999) p:05Normal>03Triage [09:33:25] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/LimboDancer was created, changed by LimboDancer link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/LimboDancer edit summary: Created page with "{{Tools Access Request |Justification=Writing small tools for my work on german wikipedia, mainly by auto-generating specific working lists inside my user name space. |Complet..." [11:02:04] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Egonw was created, changed by Egonw link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Egonw edit summary: Created page with "{{Tools Access Request |Justification=I want to set up a tool like Scholia for the upcoming Dutch elections, and later probably a similar tool for metabolites. |Completed=fals..." [13:19:21] 06Labs, 10Tool-Labs, 06Stewards-and-global-tools: For Tool Labs admins: Notice of heavy use of tool in the next month - https://phabricator.wikimedia.org/T156845#3072902 (10MarcoAurelio) Hello. SE17 are over so heavy use of this tool should not be a problem no more. Thanks. [14:54:32] 06Labs, 10Tool-Labs, 06Developer-Relations: Make sure abandoned useful tools are properly advertised so potentially interested new maintainers could find them - https://phabricator.wikimedia.org/T159595#3072963 (10Huji) [14:55:33] 06Labs, 10Tool-Labs, 06Tool-Labs-standards-committee, 06Developer-Relations, 06WMF-Legal: Make sure tools can be taken over after they are abandoned - https://phabricator.wikimedia.org/T102066#3072965 (10Huji) [15:41:22] 06Labs, 10DBA: labsdb1004 MySQL crash - https://phabricator.wikimedia.org/T159572#3073002 (10Marostegui) Looks like that database has been involved on a few things already: https://phabricator.wikimedia.org/T150759 https://phabricator.wikimedia.org/T131897 So maybe those privileges were there as a precaution... [15:50:32] 06Labs, 10Tool-Labs, 06Developer-Relations: Make sure abandoned useful tools are properly advertised so potentially interested new maintainers could find them - https://phabricator.wikimedia.org/T159595#3073003 (10zhuyifei1999) [15:52:33] 06Labs, 10Tool-Labs, 06Stewards-and-global-tools: For Tool Labs admins: Notice of heavy use of tool in the next month - https://phabricator.wikimedia.org/T156845#3073008 (10zhuyifei1999) 05Open>03Resolved [16:18:17] bd808: around? [16:34:27] Hello [17:13:55] 10Tool-Labs-tools-Zppixbot, 10Wikibugs, 10QuarryBot-enwiki: Wikibugs doesnt send phab updates to ##Zppix-Wikipedia - https://phabricator.wikimedia.org/T159612#3073166 (10Zppix) [17:18:22] PROBLEM - Puppet run on tools-exec-1412 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:49:08] I'm fixing double redirects in labs -- sorry for the RC flood [17:53:17] RECOVERY - Puppet run on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [18:01:28] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Alaa was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1636954 edit summary: [18:06:27] bd808: welcome back :) [18:53:52] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:03:57] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:12:24] 10PAWS: Enable downloading notebooks as PDF - https://phabricator.wikimedia.org/T159617#3073319 (10Tbayer) [19:28:50] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [19:43:58] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [20:54:51] PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:56:25] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Tomthirteen was modified, changed by Tomthirteen link https://wikitech.wikimedia.org/w/index.php?diff=1637290 edit summary: [21:26:10] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ranjithsiji was modified, changed by Madhuvishy link https://wikitech.wikimedia.org/w/index.php?diff=1637358 edit summary: [21:29:02] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/LimboDancer was modified, changed by Madhuvishy link https://wikitech.wikimedia.org/w/index.php?diff=1637363 edit summary: [21:29:53] RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [21:30:41] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Egonw was modified, changed by Madhuvishy link https://wikitech.wikimedia.org/w/index.php?diff=1637373 edit summary: [23:02:11] yuvipanda: worker nodes are m1.large? [23:02:20] madhuvishy: think so [23:02:37] i had it in my screen [23:02:38] yes [23:47:07] !log tools Added new k8s workers 1028, 1029 [23:47:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL