[00:01:51] 06Labs, 10Tool-Labs, 13Patch-For-Review, 07Puppet: Make standalone puppetmasters optionally use PuppetDB - https://phabricator.wikimedia.org/T153577#2958653 (10scfc) [00:01:54] 06Labs, 10Tool-Labs, 13Patch-For-Review, 07Puppet: role::puppetmaster::puppetdb depends on Ganglia and cannot be used in Labs - https://phabricator.wikimedia.org/T154104#2958652 (10scfc) 05Open>03Resolved [00:15:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:17:09] 06Labs, 10Tool-Labs, 13Patch-For-Review: Set up and use exported resources for Tool Labs's shared knowledge - https://phabricator.wikimedia.org/T153163#2871558 (10Krenair) This is what I've got in deployment-prep: ```lang=diff,name=Puppet patch commit d5bb8d6e1228b565bf92c810378e2bc6b2c1fc9a Author: Alex Mon... [00:25:52] 06Labs, 10Tool-Labs, 13Patch-For-Review: Set up and use exported resources for Tool Labs's shared knowledge - https://phabricator.wikimedia.org/T153163#2958667 (10Krenair) I prefer your approach to ssh::client. Let's split that and get it reviewed first? I have ssh::server handling whether to add an IPv6 ad... [00:57:35] 06Labs, 10Tool-Labs, 13Patch-For-Review: Set up and use exported resources for Tool Labs's shared knowledge - https://phabricator.wikimedia.org/T153163#2958708 (10Krenair) With those three patches, we're left with the role::puppetmaster::standalone thing [01:00:13] 06Labs, 10Horizon: User dschwen unable to log into horizon.wikimedia.org (An error occurred authenticating. Please try again later.) - https://phabricator.wikimedia.org/T154860#2958709 (10bd808) [01:06:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:18:14] 06Labs, 10Horizon, 07Upstream: Clicking on a project name in Identity logs you out of Horizon - https://phabricator.wikimedia.org/T138809#2958717 (10bd808) https://bugs.launchpad.net/horizon/+bug/1418246 Apparently fixed in openstack/horizon 8.0.2. We currently have verison 8.0.1-0ubuntu1~cloud0 deployed. [02:16:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:46:32] 06Labs, 10Horizon, 06Operations, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2757207 (10Krenair) It looks like what actually takes most of the time is rendering each of the 468 rows it has to display. [05:16:04] 06Labs, 10Horizon, 06Operations, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2958917 (10Krenair) It seems we can bypass all of the django/horizon template stuff going on in each row render by implementing `render` on UpdateRow in puppet_tables.py, taking off a... [05:37:03] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:36:47] PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:42:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:45:08] RECOVERY - Free space - all mounts on tools-exec-1221 is OK: OK: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [06:46:56] RECOVERY - Free space - all mounts on tools-exec-1220 is OK: OK: tools.tools-exec-1220.diskspace._public_dumps.byte_percentfree (No valid datapoints found) [07:11:48] RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:05:00] 06Labs, 10Horizon, 06Operations, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2959003 (10Joe) I would suggest, a few things: # check if the data is too large to fit on memcache (hint: they probably are). If that's the case, save them in smaller data structures.... [08:28:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:43:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:08:40] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:39:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:44:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:10:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:15:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:06:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:25:59] 10Tool-Labs-tools-LTA-Knowledgebase: Write contributing.md - https://phabricator.wikimedia.org/T155901#2959471 (10Samtar) 05Open>03Resolved a:03Samtar Merged in [[https://github.com/lta-knowledgebase/lta-tool/commit/3db112cb169f7e92cc06c06eda68066893f08423|GitHub:3db112cb169f7e92cc06c06eda68066893f08423]] [15:57:47] 10Tool-Labs-tools-LTA-Knowledgebase: Fix password hashing - https://phabricator.wikimedia.org/T155934#2959513 (10zhuyifei1999) [16:04:07] 10Tool-Labs-tools-LTA-Knowledgebase: Fix password hashing - https://phabricator.wikimedia.org/T155934#2959527 (10Samtar) p:05Triage>03High [16:06:40] 10Tool-Labs-tools-LTA-Knowledgebase: Fix password hashing - https://phabricator.wikimedia.org/T155934#2959528 (10zhuyifei1999) on a side note: you might not have to concern this if the password stuffs are redundant with OAuth. [16:22:12] 10Tool-Labs-tools-LTA-Knowledgebase: Escape against SQL injection - https://phabricator.wikimedia.org/T155935#2959530 (10zhuyifei1999) [16:24:55] 10Tool-Labs-tools-LTA-Knowledgebase: Escape against SQL injection - https://phabricator.wikimedia.org/T155935#2959543 (10Samtar) p:05Triage>03High [16:35:31] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:36:10] 10Tool-Labs-tools-LTA-Knowledgebase: Escape against XSS - https://phabricator.wikimedia.org/T155937#2959589 (10zhuyifei1999) [16:36:58] samtar [16:37:04] (yes thats me well done) [16:37:06] -.- [16:37:29] *seems like labs is having issues again? [16:37:34] 10Tool-Labs-tools-LTA-Knowledgebase: Escape against XSS - https://phabricator.wikimedia.org/T155937#2959602 (10DatGuy) p:05Triage>03High [16:39:41] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:42:35] 10Tool-Labs-tools-LTA-Knowledgebase: Add CSRF tokens - https://phabricator.wikimedia.org/T155938#2959606 (10zhuyifei1999) [16:47:08] 10Tool-Labs-tools-LTA-Knowledgebase: Limit the path of session id cookies - https://phabricator.wikimedia.org/T155940#2959633 (10zhuyifei1999) [16:51:23] 10Tool-Labs-tools-LTA-Knowledgebase: Limit the path of session id cookies - https://phabricator.wikimedia.org/T155940#2959671 (10Samtar) p:05Triage>03High [16:52:56] 10Tool-Labs-tools-LTA-Knowledgebase: Add CSRF tokens - https://phabricator.wikimedia.org/T155938#2959673 (10DatGuy) p:05Triage>03High [16:53:45] 10Tool-Labs-tools-LTA-Knowledgebase: Migrate to OAuth - https://phabricator.wikimedia.org/T155841#2959678 (10DatGuy) a:05Samtar>03DatGuy [16:53:48] 10Tool-Labs-tools-LTA-Knowledgebase: Account management module [UI] - https://phabricator.wikimedia.org/T155348#2959680 (10DatGuy) a:05Samtar>03DatGuy [16:58:57] 10Tool-Labs-tools-LTA-Knowledgebase: Add continuous integration - https://phabricator.wikimedia.org/T155944#2959711 (10DatGuy) [16:59:34] 10Tool-Labs-tools-LTA-Knowledgebase, 05Security, 07Security-Other, 07Vuln-Inject: Escape against SQL injection - https://phabricator.wikimedia.org/T155935#2959723 (10MarcoAurelio) [17:13:49] 10Tool-Labs-tools-LTA-Knowledgebase: Automatically fail authentication for any password exceeding 4096 bytes - https://phabricator.wikimedia.org/T155946#2959758 (10Samtar) [17:15:10] 10Tool-Labs-tools-LTA-Knowledgebase: Automatically fail authentication for any password exceeding 4096 bytes - https://phabricator.wikimedia.org/T155946#2959758 (10zhuyifei1999) 4096 bytes is an unnaturally-long password already. [17:15:59] 10Tool-Labs-tools-LTA-Knowledgebase: Fix password hashing - https://phabricator.wikimedia.org/T155934#2959776 (10Samtar) 05Open>03Resolved Passwords are now hashed against a strong unique and non-reused salt (generated via `openssl_random_pseudo_bytes`) per https://crackstation.net/hashing-security.htm Seco... [17:30:56] 10Tool-Labs-tools-LTA-Knowledgebase, 05Security, 07Security-Other, 07Vuln-Inject: Escape against SQL injection - https://phabricator.wikimedia.org/T155935#2959797 (10Samtar) 05Open>03Resolved [17:34:48] 10Tool-Labs-tools-LTA-Knowledgebase: Change from MD5 to bcrypt - https://phabricator.wikimedia.org/T155948#2959798 (10Samtar) [17:36:08] PROBLEM - Free space - all mounts on tools-exec-1221 is CRITICAL: CRITICAL: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-exec-1221.diskspace.root.byte_percentfree (<22.22%) [18:02:51] 10Tool-Labs-tools-LTA-Knowledgebase: Escape against XSS - https://phabricator.wikimedia.org/T155937#2959841 (10Samtar) 05Open>03Resolved Tracked down all the displays and escaped them - confirmed this was an issue beforehand and confirmed it is resolved [18:19:44] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [20:16:02] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:33:57] (03Draft1) 10Paladox: Add project phabricator-* [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333531 [21:34:00] (03PS2) 10Paladox: Add project phab-* [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333531 [21:35:48] (03Draft1) 10Paladox: Add the phabricator and phab-* project o #wikimedia-releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333539 [21:35:50] (03PS2) 10Paladox: Add the phabricator and phab-* project o #wikimedia-releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333539 [21:36:11] (03CR) 10jerkins-bot: [V: 04-1] Add the phabricator and phab-* project o #wikimedia-releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333539 (owner: 10Paladox) [21:37:35] (03PS3) 10Paladox: Add the phabricator and phab-* project o #wikimedia-releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333539 [21:38:28] (03PS4) 10Paladox: Add the phabricator and phab-* project o #wikimedia-releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333539 [21:59:55] (03Draft1) 10Paladox: Add phabricator-upstream to #wikimedia-dev and #wikimedia-devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333553 [21:59:57] (03PS2) 10Paladox: Add phabricator-upstream to #wikimedia-dev and #wikimedia-devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333553 [22:07:00] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]