[01:20:19] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Sealnk was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=155622 edit summary: [03:26:39] PROBLEM - Puppet failure on tools-webgrid-07 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [03:51:39] RECOVERY - Puppet failure on tools-webgrid-07 is OK: OK: Less than 1.00% above the threshold [0.0] [05:29:02] Anyone around? [05:30:00] Er nvm [05:52:20] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Triviallab was created, changed by Triviallab link https://wikitech.wikimedia.org/wiki/Nova+Resource%3aTools%2fAccess+Request%2fTriviallab edit summary: Created page with "{{Tools Access Request |Justification=I don't know... |Completed=false |User Name=Triviallab }}" [06:32:46] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:34:35] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:36:43] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:01:42] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [07:02:44] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:04:36] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [08:32:14] (03PS1) 10Nemo bis: Script to produce #target list of users for input languages [labs/tools/lists] - 10https://gerrit.wikimedia.org/r/206630 (https://phabricator.wikimedia.org/T90536) [08:33:19] 10Tool-Labs-tools-Other, 5Patch-For-Review: Script to produce #target list of users for input languages - https://phabricator.wikimedia.org/T90536#1236510 (10Nemo_bis) 5Open>3Resolved I ended up doing something slightly more polished than a shell script. I also added it somewhere in gerrit for the archives... [08:55:20] 10Tool-Labs: webservice start / restart shouldn't always need to specify type of server - https://phabricator.wikimedia.org/T97245#1236541 (10yuvipanda) 3NEW [09:08:51] RECOVERY - Puppet failure on tools-bastion-01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:44:53] PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:11:48] 10Wikimedia-Labs-Infrastructure: Make the link to list of all Labs projects more visible - https://phabricator.wikimedia.org/T53889#1236602 (10Aklapper) a:5RyanLane>3None [12:11:52] 10MediaWiki-extensions-OpenStackManager, 5Patch-For-Review: Strip linebreaks in uploaded ssh keys - https://phabricator.wikimedia.org/T51553#1236600 (10Aklapper) a:5RyanLane>3None [12:12:25] 10Wikimedia-Labs-Infrastructure: accommodate project-specific puppet git branches - https://phabricator.wikimedia.org/T37737#1236634 (10Aklapper) a:5RyanLane>3None [12:12:26] 10Wikimedia-Labs-Infrastructure: Change network configuration to network-node per compute-node - https://phabricator.wikimedia.org/T48819#1236641 (10Aklapper) a:5RyanLane>3None [13:09:50] RECOVERY - Puppet failure on tools-bastion-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:45:52] PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:39:37] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:42:45] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:54:38] 6Labs: Labs puppetmaster says: Could not resolve 10.68.16.210: no name for 10.68.16.210 - https://phabricator.wikimedia.org/T97250#1236831 (10Andrew) 3NEW [15:07:45] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:14:37] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:11:00] 10Tool-Labs: toolsbeta: set up puppet-compiler / temporary-apply - https://phabricator.wikimedia.org/T97081#1236901 (10valhallasw) Thanks, that's a super useful explanation. As I understand it, puppet-compiler should be able to give the relevant diffs (although not the actual diffs, but the compiled-manifest-to... [17:12:37] (03CR) 10Alessio: "Please, can you explain why to add this .py script to lists?" [labs/tools/lists] - 10https://gerrit.wikimedia.org/r/206630 (https://phabricator.wikimedia.org/T90536) (owner: 10Nemo bis) [17:12:42] !log toolsbeta deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test [17:12:46] Logged the message, Master [17:13:42] (03CR) 10Nemo bis: "Well, it makes a list and the first use was for it.wiki. :) It would be cute to share queries etc. in gerrit, IMHO." [labs/tools/lists] - 10https://gerrit.wikimedia.org/r/206630 (https://phabricator.wikimedia.org/T90536) (owner: 10Nemo bis) [17:33:43] (03CR) 10Alessio: "So your idea is to use this tool's backend to store also some scripts (without frontend integration) that belong to the "SQL query world"?" [labs/tools/lists] - 10https://gerrit.wikimedia.org/r/206630 (https://phabricator.wikimedia.org/T90536) (owner: 10Nemo bis) [17:38:26] (03CR) 10Nemo bis: "Well, it could be a subdirectory of public_html if you want it visible. I don't have any complex idea here, I just looked for an existing " [labs/tools/lists] - 10https://gerrit.wikimedia.org/r/206630 (https://phabricator.wikimedia.org/T90536) (owner: 10Nemo bis) [17:41:53] (03CR) 10Alessio: [C: 031 V: 031] "I agree probably this is actually the best fit. We can start with an hidden dir, remembering the possibility to add complexity in the futu" [labs/tools/lists] - 10https://gerrit.wikimedia.org/r/206630 (https://phabricator.wikimedia.org/T90536) (owner: 10Nemo bis) [17:43:53] 10MediaWiki-extensions-OpenStackManager: OpenStackManager special pages should link to in-wiki documentation - https://phabricator.wikimedia.org/T36500#1236918 (10Nemo_bis) So, is there any documentation we can link? [18:18:35] !log toolsbeta having some issues with puppet-test, so postponing for now [18:18:39] Logged the message, Master [19:09:22] 10Wikimedia-Labs-Infrastructure: e-mail sending from labs - https://phabricator.wikimedia.org/T38511#1237020 (10hashar) [19:09:23] 10Wikimedia-Labs-Infrastructure: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#1237021 (10hashar) [19:11:55] 10Wikimedia-Labs-Infrastructure: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#1237028 (10hashar) Labs still route through prod servers: ``` $ exim -bt foo@example.org foo@example.org router = smart_route, transport = remote_smtp host polonium.wikimedia.org [2620:0:861:3:208:80:154:... [19:14:33] 6Labs: /etc/ssh/userkeys/ubuntu notices for every puppet run on labs instances - https://phabricator.wikimedia.org/T94866#1237029 (10hashar) [19:14:35] 6Labs: /etc/ssh/userkeys/ubuntu notices for every puppet run on labs instances - https://phabricator.wikimedia.org/T94866#1174750 (10hashar) Related / dupes: {T59752} {T81099} {T85814} [19:30:53] RECOVERY - Puppet failure on tools-bastion-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:01:52] PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:42:19] "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter settings" yes thank you puppet, that's very informative :{ [20:50:18] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Triviallab was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=155672 edit summary: [21:37:27] 6Labs, 10LabsDB-Auditor, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, and 10 others: Labs' Phabricator tags overhaul - https://phabricator.wikimedia.org/T89270#1237124 (10Aklapper) >>! In T89270#1032542, @Legoktm wrote: > Wikimedia-Labs-extdist is for bugs about the "extdist" project in labs (extd... [22:30:48] PROBLEM - Puppet failure on tools-static-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:15:47] RECOVERY - Puppet failure on tools-static-02 is OK: OK: Less than 1.00% above the threshold [0.0]