[00:16:53] andrewbogott ^ [00:18:37] jaufrecht: I'll look [00:19:04] jaufrecht: what is the actual name of the instance? [00:27:21] 06Labs: Labs instances failing with "internal error: No PCI buses available" - https://phabricator.wikimedia.org/T137857#2381540 (10Andrew) [00:27:37] 06Labs: Labs instances failing with "internal error: No PCI buses available" - https://phabricator.wikimedia.org/T137857#2381553 (10Andrew) [00:27:39] 06Labs, 10Phlogiston (Interrupt), 15User-bd808: Phlogiston-1 unresponsive - https://phabricator.wikimedia.org/T137736#2381554 (10Andrew) [00:35:55] 06Labs: Labs instances failing with "internal error: No PCI buses available" - https://phabricator.wikimedia.org/T137857#2381564 (10Andrew) [00:35:57] 06Labs, 10Phlogiston (Interrupt), 15User-bd808: Phlogiston-1 unresponsive - https://phabricator.wikimedia.org/T137736#2381562 (10Andrew) 05Open>03Resolved I migrated this VM to labvirt1009 and did a bit of puppet/apt cleanup, and it seems to be working fine. The original cause of the lockup is unknown,... [00:36:08] jaufrecht: undiagnosed, but fixed. Sorry about the downtime. [01:22:42] Thanks. [03:59:30] 06Labs, 10Phabricator: https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error - https://phabricator.wikimedia.org/T137270#2381698 (10mmodell) I can't log in to any of the phab hosts so I don't know what to do to help. [04:19:28] 06Labs, 10Labs-Infrastructure, 10DBA: Add shared edit rights between eranbot and copypatrol tools - https://phabricator.wikimedia.org/T137691#2381717 (10Niharika) >>! In T137691#2380490, @kaldari wrote: > This is on the toolsdb. Probably doesn't need any work from you, @jcrespo. It does. See {T134392} [04:31:05] 06Labs, 10Labs-Infrastructure, 10DBA: Add shared edit rights between eranbot and copypatrol tools - https://phabricator.wikimedia.org/T137691#2381723 (10kaldari) My bad :) [05:07:18] 06Labs, 10Phabricator: https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error - https://phabricator.wikimedia.org/T137270#2363140 (10yuvipanda) I just rebooted phab-01 and can log in to it. Try doing the same for the rest? [05:14:27] 06Labs, 10Phabricator: https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error - https://phabricator.wikimedia.org/T137270#2381734 (10yuvipanda) I rebooted -02 as well, but that doesn't let me in - not user key nor root key. Same for -04. I pronounce those two as dead, and -01 and -0... [05:26:31] Krenair: thanks! [05:28:31] 06Labs, 10Phabricator: https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error - https://phabricator.wikimedia.org/T137270#2381745 (10yuvipanda) I actually looked a bit deeper into phab-02, it turns out the problems were manyfold: 1. It had a puppetmaster set to 'deploy.phabricator.... [08:07:28] 06Labs, 10Labs-Infrastructure, 10DBA: Add shared edit rights between eranbot and copypatrol tools - https://phabricator.wikimedia.org/T137691#2382000 (10jcrespo) 05Open>03Resolved The following was executed on labsdb1001: ``` GRANT ALL PRIVILEGES ON `s51306\_\_copyright\_p`.* TO 's53018'@'%'; ``` That... [08:59:33] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ani3011 was created, changed by Ani3011 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Ani3011 edit summary: Created page with "{{Tools Access Request |Justification=I want to create a project on Economics. |Completed=false |User Name=Ani3011 }}" [09:16:21] 06Labs, 10Phabricator: https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error - https://phabricator.wikimedia.org/T137270#2382097 (10Paladox) @mmodell would you be able to upgrade phabricator on phab-01 and phab-03 please. [09:20:47] 06Labs, 10Phabricator: https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error - https://phabricator.wikimedia.org/T137270#2382103 (10Paladox) @yuvipanda thanks for fixing it. [09:58:51] !ping [09:58:52] !pong [12:17:12] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ani3011 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=655381 edit summary: [16:02:20] 06Labs, 10Tool-Labs, 10DBA, 06Operations, 10Traffic: Antigng-bot improper non-api http requests - https://phabricator.wikimedia.org/T137707#2382892 (10Antigng_) >>! In T137707#2380275, @jcrespo wrote: > BTW, the API is definitely faster, one just need to use it efficiently: > > > ``` > $ time curl 'htt... [16:06:09] (03PS1) 10Andrew Bogott: Added dummy password for labspuppetbackend::mysql_password: [labs/private] - 10https://gerrit.wikimedia.org/r/294516 [16:06:25] (03CR) 10Andrew Bogott: [C: 032 V: 032] Added dummy password for labspuppetbackend::mysql_password: [labs/private] - 10https://gerrit.wikimedia.org/r/294516 (owner: 10Andrew Bogott) [16:06:57] 06Labs, 10Tool-Labs, 10DBA, 06Operations, 10Traffic: Antigng-bot improper non-api http requests - https://phabricator.wikimedia.org/T137707#2382924 (10Antigng_) >>! In T137707#2379997, @jcrespo wrote: > For the API part, I would like to add that API infrastructure (application servers and databases) is s... [16:14:26] (03PS1) 10Andrew Bogott: Add labspuppetbackend::mysql_password to a few more files [labs/private] - 10https://gerrit.wikimedia.org/r/294518 [16:14:41] (03CR) 10Andrew Bogott: [C: 032 V: 032] Add labspuppetbackend::mysql_password to a few more files [labs/private] - 10https://gerrit.wikimedia.org/r/294518 (owner: 10Andrew Bogott) [16:23:28] Hi [16:23:37] I'm pretty new to Tool Labs [16:24:05] But I wanted to try using ToolLabs to run some MySQL queries on the enWiki database [16:24:25] Is there any tutorial available or something which helps outline the steps for the same? [16:27:17] SoniWP: There is a tool called Quarry that can run queriues against replica databases. [16:27:31] https://quarry.wmflabs.org/ [16:27:42] It should have plenty of examples floating around. [16:27:48] Have you ever done SQL before? [16:28:14] Matthew_, Yes, I am aware of that thank you. But Quarry does not support too extensive queries. I ran a few SQL queries that got killed by Quarry for timeout no matter how I optimise them [16:28:41] And yes, I have done SQL before to have a decent understanding of things [16:28:44] Unfortunetly the timeout is a tools-wide thing - it's actually enforced by the SQL server as far as I know. [16:29:19] This is the type of queries I intend to run [16:29:20] https://quarry.wmflabs.org/query/10293 [16:29:51] Hm. [16:29:55] User:Halfak suggested me to use Tool Labs to run them because of the Quarry timeout. [16:30:30] What is the purpose of this query? [16:30:30] (I assume he has done similar queries before so he might know about Quarry's limitations on this_) [16:30:35] https://phabricator.wikimedia.org/T135644 [16:30:42] This is the task I'm working on. [16:31:01] Ah. [16:31:11] Okay. I follow now. [16:31:16] So do you have a tool? [16:31:23] As in? [16:32:10] For tool labs, everything has to run as a tool account. That's where you get your database credentials. [16:32:20] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Creating_a_new_Tool_account [16:32:49] There is a replica.my.cnf file in the root of your tool account. This has your database credentials. From there, just create a script and run it. [16:32:57] Does that make sense? [16:33:38] Not completely, but sounds like something I can try working on. Thanks. [16:34:16] Sure. I'll be around for a bit - and while I don't work for the WMF, I have been around the block with this a few times. So feel free to poke if no-one is around. [16:34:23] Sure, thanks. [16:34:43] P.S. I am at the "Run ssh" step of https://www.mediawiki.org/wiki/Gerrit/Tutorial#Generate_a_new_SSH_key [16:35:02] And get this error - ssh: connect to host gerrit.wikimedia.org port 29418: Network is unreachable [16:35:33] Is this something on my side that I configured badly or is it some sort of permissions problem? [16:35:38] Matthew_, ^ [16:36:15] Hm. Sounds like something on your end... Why are you going to Gerrit? [16:36:58] Um, I'm not quite sure, honestly. I was just following the steps to generate a new SSH key [16:37:10] Ah. [16:37:16] I assumed SSHing into gerrit would be part of it to check if it works [16:38:16] Hm. [16:40:04] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Access Try this. [16:47:33] Oh. It seems like I do not have access to the Tools project. (I assumed it was bundled with/similar to SHell access) [16:47:42] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Soni was created, changed by Soni link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Soni edit summary: Created page with "{{Tools Access Request |Justification=https://phabricator.wikimedia.org/T135644 |Completed=false |User Name=Soni }}" [16:48:53] That would explain it. [16:53:42] SoniWP: as far as I know, gerrit keys are unrelated to labs/tools ssh keys [16:53:59] maybe you've got that sorted by now :) [17:03:56] andrewbogott, Honestly I still dont completely understand what is going on, but thanks [17:04:46] SoniWP: this might be a reasonable place to start: https://wikitech.wikimedia.org/wiki/Portal:Tool_Labs [17:39:32] !log git-phab-3 upgrading phabricator, libphutil and arcanist to master branch. [17:39:33] git-phab-3 is not a valid project. [17:40:31] !log git upgrading git-phab-3 instances phabricator, libphutil and arcanist to master branch. [17:40:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL, Master [17:47:18] all the Beta Labs wikis have been database locked for the past half hour or so [17:48:09] kaldari: past 12 hours or so in fact [17:48:35] kaldari #wikimedia-releng are aware and are trying to fix it. [17:48:57] cool [18:04:01] 06Labs, 10Phabricator: https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error - https://phabricator.wikimedia.org/T137270#2383301 (10mmodell) I fixed up phab-03 and upgraded it. I'll work on phab-01 next [18:09:54] andrewbogott, Thanks. [18:10:13] Also, would you know how long would it take before I get access to the Toold project? [18:10:29] 06Labs, 10Phabricator: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#2383313 (10mmodell) [18:10:33] 06Labs, 10Phabricator: https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error - https://phabricator.wikimedia.org/T137270#2383310 (10mmodell) 05Open>03Resolved a:03yuvipanda thanks @yuvipanda [18:24:20] 06Labs, 10Phabricator: https://phab-01.wmflabs.org (and other installs, like phab-02) returns a 502 error - https://phabricator.wikimedia.org/T137270#2383366 (10Krenair) Is phab-04 repairable via salt/changing roles in the same way phab-02 was? [18:37:16] SoniWP: not long if you ask me here — did you file a request already? [19:08:38] andrewbogott, Yes. https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Soni [19:10:01] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Soni was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=656613 edit summary: [19:50:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Soni was modified, changed by Andrew Bogott link https://wikitech.wikimedia.org/w/index.php?diff=656728 edit summary: [20:40:57] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2383843 (10yuvipanda) [20:57:00] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2383930 (10hashar) The resolv.conf for different instances: Nodepool... [20:58:20] this is a test? [20:58:22] ah ok [21:13:50] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2383969 (10RobH) Since this is likely to happen when others are at wikimania, can you list the exact data directories that require copy? [21:17:06] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2383972 (10yuvipanda) [21:18:30] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2383979 (10hashar) I have looked at a Jessie Nodepool instance. It ha... [21:31:39] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2383987 (10hashar) In puppet we have modules/openstack/templates/libe... [21:32:54] 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2383988 (10hashar) Forgot, a fully commented dnsmasq.conf http://www.... [22:08:34] 06Labs, 10Labs-Infrastructure: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2384072 (10yuvipanda)