[00:16:08] 06Labs, 10Tool-Labs: Create NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2932693 (10scfc) 05Open>03Invalid NodeJS containers already exist; cf. https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#node.js_web_services. [03:11:12] 06Labs, 10Tool-Labs, 10DBA: Credentials in replica.my.cnf gives Access Denied - https://phabricator.wikimedia.org/T154933#2932929 (10scfc) [03:13:34] 06Labs, 10Tool-Labs, 10DBA: Credentials in replica.my.cnf gives Access Denied - https://phabricator.wikimedia.org/T154933#2928837 (10scfc) (`user` was `('s51488',)` in `/data/project/danmicholobot/replica.my.cnf`; I fixed that to `s51488`, but it does not change the access problem.) [03:14:48] 06Labs, 10Tool-Labs, 10DBA: Credentials in replica.my.cnf gives Access Denied - https://phabricator.wikimedia.org/T154933#2932936 (10scfc) [03:52:11] 06Labs, 10Tool-Labs, 10DBA: Credentials in replica.my.cnf gives Access Denied - https://phabricator.wikimedia.org/T154933#2928837 (10Marostegui) The grants are there... ``` [root@labsdb1001 03:48 /root] # pt-show-grants | grep s51488 -- Grants for 's51488'@'%' GRANT SELECT, SHOW VIEW ON `%\_p`.* TO... [05:46:48] !log video re-enabling v2c frontend & pooling worker at encoding02; transcode queue is < 500 now, and mostly containing relatively small files (T153488) [05:46:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [05:46:51] T153488: Commons video transcoders have over 6500 tasks in the backlog. - https://phabricator.wikimedia.org/T153488 [06:45:19] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:47:30] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:51:49] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Quiddity was created, changed by Quiddity link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Quiddity edit summary: Created page with "{{Tools Access Request |Justification=Learn the interfaces, take screenshots, and poke at the language used in various tools. |Completed=false |User Name=Quiddity }}" [06:54:34] (03CR) 10Lokal Profil: [C: 031] Refactor database_statistics.getStatistics (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/331407 (owner: 10Jean-Frédéric) [07:01:10] PROBLEM - Puppet run on tools-exec-1419 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:08:06] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:22:28] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [07:25:20] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [07:26:44] (03PS3) 10Lokal Profil: Filter out blacklisted categories during categorisation [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/328348 (https://phabricator.wikimedia.org/T153744) (owner: 10Jean-Frédéric) [07:28:06] (03CR) 10Lokal Profil: "patched to make this work with the hidden categories patch. also changed the cat in the yml to a non-hidden one" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/328348 (https://phabricator.wikimedia.org/T153744) (owner: 10Jean-Frédéric) [07:36:09] RECOVERY - Puppet run on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [07:43:07] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [08:18:57] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:28:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:58:59] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [09:03:04] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:47:02] PROBLEM - Free space - all mounts on tools-worker-1003 is CRITICAL: CRITICAL: tools.tools-worker-1003.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found) tools.tools-worker-1003.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-worker-1003.diskspace.root.byte_percentfree (<100.00%) [13:59:59] 06Labs, 10Tool-Labs, 10DBA: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2933264 (10akosiaris) @Tobias1984. Sorry I completely forgot about this. I 've created your account. Contact me privately on IRC (akosiaris) or via email so I can provide the credentials to you. [14:40:52] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team: E-Mail Adress - https://phabricator.wikimedia.org/T155093#2933319 (10Katja_Ullrich_WMDE) [15:25:58] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Methaspirin was modified, changed by Methaspirin link https://wikitech.wikimedia.org/w/index.php?diff=1301263 edit summary: [15:47:14] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Methaspirin was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1301399 edit summary: [16:18:10] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Quiddity was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1301940 edit summary: [17:21:21] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: labtest mwyaml hiera backend causes errors - https://phabricator.wikimedia.org/T145808#2933588 (10Krenair) a:03Krenair [17:36:01] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933601 (10scfc) [17:36:47] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2928837 (10scfc) [17:48:26] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933613 (10scfc) Some more and command to list them: ``` scfc@tools-bastion-03:~$ sudo find -H /home /data/project -mindepth 2 -maxdepth 2 -type f -name replica.my.cnf -... [18:01:37] !log deployment-prep disabling puppet on restbase cassandra nodes to experiment with prometheus exporter [18:01:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:07:45] !log deployment-prep restarting restbase cassandra nodes [18:07:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:12:51] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933700 (10Marostegui) Do you happen to know to which of the labsdb servers are you connecting to? [18:26:48] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933752 (10scfc) That's probably `10.64.4.11`: ``` scfc@tools-bastion-03:~$ host enwiki.labsdb enwiki.labsdb has address 10.64.4.11 scfc@tools-bastion-03:~$ ``` [18:29:44] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2928837 (10chasemp) Known broken AFAIK: tools.jackbot tools.danmicholobot tools.hasteurbot tools.reportsbot tools.enbbsb tools.botomatik tools.hgztools tools.xxn-tools t... [18:30:38] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933779 (10Marostegui) >>! In T154933#2933752, @scfc wrote: > That's probably `10.64.4.11`: > > ``` > scfc@tools-bastion-03:~$ host enwiki.labsdb > enwiki.labsdb has add... [18:34:28] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933781 (10chasemp) ```root@labstore1004:~# nano /tmp/bad-tool-labsdb root@labstore1004:~# for tool in `cat /tmp/bad-tool-labsdb`; do /usr/local/sbin/maintain-dbusers del... [18:36:27] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933783 (10Marostegui) @scfc can you try those ones that Chase recreated and see if they work? [18:38:23] 06Labs, 10Striker, 10Horizon, 10wikitech.wikimedia.org: Wikitech/horizon/striker 2fa documentation - https://phabricator.wikimedia.org/T155108#2933797 (10Andrew) [18:38:47] (03CR) 10Andrew Bogott: [C: 032] Check and enforce OATH account protection (032 comments) [labs/striker] - 10https://gerrit.wikimedia.org/r/327786 (https://phabricator.wikimedia.org/T144712) (owner: 10BryanDavis) [18:40:23] (03Merged) 10jenkins-bot: Check and enforce OATH account protection [labs/striker] - 10https://gerrit.wikimedia.org/r/327786 (https://phabricator.wikimedia.org/T144712) (owner: 10BryanDavis) [18:41:03] (03CR) 10Andrew Bogott: [C: 032] Combine account settings screens [labs/striker] - 10https://gerrit.wikimedia.org/r/328056 (owner: 10BryanDavis) [18:43:57] (03Merged) 10jenkins-bot: Combine account settings screens [labs/striker] - 10https://gerrit.wikimedia.org/r/328056 (owner: 10BryanDavis) [18:44:11] (03CR) 10Andrew Bogott: [C: 032] Add FontAwesome css, fonts, and templatetag helper [labs/striker] - 10https://gerrit.wikimedia.org/r/328057 (owner: 10BryanDavis) [18:51:19] (03Merged) 10jenkins-bot: Add FontAwesome css, fonts, and templatetag helper [labs/striker] - 10https://gerrit.wikimedia.org/r/328057 (owner: 10BryanDavis) [18:53:31] (03CR) 10Andrew Bogott: [C: 032] Display existing SSH keys [labs/striker] - 10https://gerrit.wikimedia.org/r/328058 (https://phabricator.wikimedia.org/T144711) (owner: 10BryanDavis) [18:55:45] (03Merged) 10jenkins-bot: Display existing SSH keys [labs/striker] - 10https://gerrit.wikimedia.org/r/328058 (https://phabricator.wikimedia.org/T144711) (owner: 10BryanDavis) [19:09:18] 06Labs, 10MediaWiki-Vagrant, 13Patch-For-Review, 07Puppet, 15User-bd808: Make role::labs::mediawiki_vagrant work on Debian Jessie host systems - https://phabricator.wikimedia.org/T154340#2933915 (10bd808) [19:21:36] 06Labs, 10Striker, 10Horizon, 10wikitech.wikimedia.org, 07Documentation: Wikitech/horizon/striker 2fa documentation - https://phabricator.wikimedia.org/T155108#2933945 (10bd808) p:05Triage>03High [19:22:45] 06Labs, 10Striker, 10Horizon, 10wikitech.wikimedia.org, 07Documentation: Wikitech/horizon/striker 2fa documentation - https://phabricator.wikimedia.org/T155108#2933797 (10bd808) If you can get a stub started I'd be glad to help fill in some bits and bobs. [19:25:15] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2933950 (10scfc) The credentials for the 13 tools now work: ``` scfc@tools-bastion-03:~$ for tool in alex botomatik boty danmicholobot enbbsb hasteurbot hgztools jackbot... [19:27:13] 06Labs, 10Tool-Labs: Create NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2933954 (10Tarrow) Sorry; I'd obviously overlooked that there is currently a very old version of NodeJS available. Would it be possible to have a newer version? The version available is 0.10.x but the curren... [19:28:34] 06Labs, 10Tool-Labs: Create Updated NodeJS container for Tool Labs - https://phabricator.wikimedia.org/T155063#2933958 (10Tarrow) 05Invalid>03Open [19:37:29] PROBLEM - Puppet run on tools-static-11 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:38:49] PROBLEM - Puppet run on tools-worker-1013 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:39:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:39:45] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:41:07] PROBLEM - Puppet run on tools-worker-1023 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:14:44] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:16:07] RECOVERY - Puppet run on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [20:17:29] RECOVERY - Puppet run on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0] [20:18:49] RECOVERY - Puppet run on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:09:00] 06Labs, 10Tool-Labs, 10DBA: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2934226 (10Marostegui) The user `danmichaelo` isn't in the list that @chasemp iterated over the first time to test if that would fix the issue. Looks like recreating that... [21:36:14] (03CR) 10Andrew Bogott: [C: 032] "Bonus points for the strident delete confirmation :)" [labs/striker] - 10https://gerrit.wikimedia.org/r/328117 (https://phabricator.wikimedia.org/T144711) (owner: 10BryanDavis) [21:37:45] (03Merged) 10jenkins-bot: Allow deleting SSH keys [labs/striker] - 10https://gerrit.wikimedia.org/r/328117 (https://phabricator.wikimedia.org/T144711) (owner: 10BryanDavis) [21:45:02] (03CR) 10Andrew Bogott: Add SSH key importing (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/328403 (https://phabricator.wikimedia.org/T144711) (owner: 10BryanDavis) [22:09:19] !log add Reedy to admin in tool labs (approved by bryon and chase for access to investigate specific tool abuse behavior) [22:09:19] Unknown project "add" [22:09:23] !log tools add Reedy to admin in tool labs (approved by bryon and chase for access to investigate specific tool abuse behavior) [22:09:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:19:57] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:26:59] Earwig: for copyvio detector i think it could be useful to show all results in one go like https://www.kent.ac.uk/ai/staff/usingturnitin/images/intor12.gif ; and provide context from source surrounding the matching part, in case the user copied some text over _and_ scuffed up some other text which remained undetected [22:54:57] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [22:56:55] (03CR) 10Andrew Bogott: [C: 032] "Update: Bryan confirms that it will reject private keys, although without an explanation. Good enough for now." [labs/striker] - 10https://gerrit.wikimedia.org/r/328403 (https://phabricator.wikimedia.org/T144711) (owner: 10BryanDavis) [22:58:53] (03Merged) 10jenkins-bot: Add SSH key importing [labs/striker] - 10https://gerrit.wikimedia.org/r/328403 (https://phabricator.wikimedia.org/T144711) (owner: 10BryanDavis) [22:58:55] (03Merged) 10jenkins-bot: Cleanup a few things with account registration form [labs/striker] - 10https://gerrit.wikimedia.org/r/328620 (owner: 10BryanDavis) [22:59:21] (03Merged) 10jenkins-bot: Remove some vertical whitespace on linked accounts screen [labs/striker] - 10https://gerrit.wikimedia.org/r/328621 (owner: 10BryanDavis) [22:59:24] (03Merged) 10jenkins-bot: Allow changing LDAP password [labs/striker] - 10https://gerrit.wikimedia.org/r/328622 (https://phabricator.wikimedia.org/T153935) (owner: 10BryanDavis) [23:00:05] (03Merged) 10jenkins-bot: Point SSH key goal at local key management screen [labs/striker] - 10https://gerrit.wikimedia.org/r/328841 (https://phabricator.wikimedia.org/T144711) (owner: 10BryanDavis) [23:00:44] (03Merged) 10jenkins-bot: Add a password strength meter [labs/striker] - 10https://gerrit.wikimedia.org/r/329018 (https://phabricator.wikimedia.org/T153935) (owner: 10BryanDavis) [23:32:49] 10Tool-Labs-tools-Erwin's-tools, 06Stewards-and-global-tools: Restore delete.php - https://phabricator.wikimedia.org/T107663#2934869 (10MarcoAurelio) [23:44:08] 06Labs, 10Striker, 10Horizon, 10wikitech.wikimedia.org, 07Documentation: Wikitech/horizon/striker 2fa documentation - https://phabricator.wikimedia.org/T155108#2935230 (10bd808) I started https://wikitech.wikimedia.org/wiki/Help:Two-factor_authentication