[02:10:02] PROBLEM - SSH on tools-merlbot-proxy is CRITICAL: Server answer [02:15:02] RECOVERY - SSH on tools-merlbot-proxy is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [04:42:32] Until I find the old topic to set it back [06:08:33] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Zppix was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=808378 edit summary: [06:48:06] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 531 bytes in 0.027 second response time [06:53:05] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.027 second response time [10:42:50] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:22:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:26:25] legoktm, bd808, I think I have fixed gerrit-reviewer-bot again. The pywikibot version used was super out of date and /that/ was causing issues as well [11:26:54] valhallasw`cloud hi, did you deploy it. [11:27:02] please [11:27:02] ? [11:27:03] ... [11:27:12] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:28:10] oh so you did? [11:29:03] valhallasw`cloud ^^ [11:44:58] valhallasw`cloud Testing the reviewer bot here https://gerrit.wikimedia.org/r/#/c/302082/ dosent seem to add hashar [11:45:10] Does the reviewer bot also support gerrits new on the web editor. [11:45:12] ? [11:45:49] paladox: there's a backlog that still needs to be processed [11:45:56] Oh [11:45:58] thanks [11:46:18] paladox: and in general it only runs every five minutes, so a test that was created three minutes ago is not a very good one [11:46:44] valhallasw`cloud sorry for another ping, but do you see the queue being processed. [11:46:50] and ok [11:47:32] yay it works [11:47:34] i got the email [11:50:23] 10Labs-Other-Projects, 10Gerrit: gerrit-reviewer-bot stopped working - https://phabricator.wikimedia.org/T141390#2509403 (10Paladox) Fixed by @valhallasw [11:50:47] 10Labs-Other-Projects, 10Gerrit: gerrit-reviewer-bot stopped working - https://phabricator.wikimedia.org/T141390#2509404 (10Paladox) 05Open>03Resolved a:03valhallasw Closing as resolved. [12:00:43] valhallasw`cloud are you sure it supports the on the web gerrit editor or does it only work when git pushing from a git client. [12:00:44] ? [12:01:07] If Gerrit sends out an e-mail, it should work. [12:01:30] Ok [12:12:49] valhallasw`cloud it seems the first patchset is made a draft would that stop the reviewer bot from working on that patch even if you publish it. [12:13:02] for example [12:13:03]

(Draft2) Paladox: Testing reviewer bot [integration/config] - https://gerrit.wikimedia.org/r/302082 [12:13:03] * aharoni (55fad010@gateway/web/freenode/ip.85.250.208.16) has joined [12:13:04]

(PS3) Paladox: Testing reviewer bot [integration/config] - https://gerrit.wikimedia.org/r/302082 [12:13:04]

(Draft1) Paladox: Testing reviewer bot [integration/config] - https://gerrit.wikimedia.org/r/302082 [12:13:06]

(Abandoned) Paladox: Testing reviewer bot [integration/config] - https://gerrit.wikimedia.org/r/302082 (owner: Paladox) [12:13:11]

(Draft1) Paladox: Testing reviewer bot [integration/config] - https://gerrit.wikimedia.org/r/302083 [12:13:14]

(PS2) Paladox: Testing reviewer bot [integration/config] - https://gerrit.wikimedia.org/r/302083 [12:13:17] Never mind [12:13:19] it works [12:13:23] Sorry for the ping again [12:13:47] paladox: for all of these questions: test it, or read the code [12:13:58] Ok [12:35:33] Hi there. Hopefully somebody around for this simple question [12:35:46] Trying to connect to Redis on Labs at localhost:6379 [12:36:18] But doesn't seem to be there. Where can I find it? [12:37:16] tools-redis afair [12:37:26] if it is tools [12:42:13] Yes, seems to do the trick. Thanks a lot :) [12:47:13] np :) [13:31:04] PROBLEM - SSH on tools-merlbot-proxy is CRITICAL: Server answer [13:46:02] RECOVERY - SSH on tools-merlbot-proxy is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [13:52:02] PROBLEM - SSH on tools-merlbot-proxy is CRITICAL: Server answer [14:08:27] 06Labs, 10Tool-Labs: Puppet disabled on tools-worker-10* - https://phabricator.wikimedia.org/T141719#2509551 (10valhallasw) [14:09:23] 06Labs, 10Tool-Labs: Puppet disabled on tools-worker-10* - https://phabricator.wikimedia.org/T141719#2509557 (10valhallasw) Same on tools-docker-builder-01. [14:13:40] 06Labs, 10Tool-Labs: tools-docker-builder-03 hanging - https://phabricator.wikimedia.org/T141720#2509565 (10valhallasw) [14:14:39] 06Labs: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2507274 (10valhallasw) Can you clarify what the columns mean? tools-merlbot-proxy is currently hanging with https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&instanceid=7b12cce8-b045-4fed-96... [14:16:11] 06Labs, 10Tool-Labs: tools-webgrid-lighttpd-1203: toollabs-webservice : Depends: python-pykube but it is not going to be installed - https://phabricator.wikimedia.org/T141721#2509594 (10valhallasw) [14:17:38] 06Labs, 10Tool-Labs: tools-webgrid-lighttpd-1203: toollabs-webservice : Depends: python-pykube but it is not going to be installed - https://phabricator.wikimedia.org/T141721#2509596 (10valhallasw) [14:18:07] 06Labs, 10Tool-Labs: tools-webgrid-lighttpd-1203: toollabs-webservice : Depends: python-pykube but it is not going to be installed - https://phabricator.wikimedia.org/T141721#2509583 (10valhallasw) 05Open>03Resolved a:03valhallasw `sudo apt-get -f install` suggested to install `python-pykube`, which seem... [14:32:14] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [14:37:02] RECOVERY - SSH on tools-merlbot-proxy is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [15:38:12] 06Labs: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2509688 (10yuvipanda) [15:38:30] 06Labs: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2507274 (10yuvipanda) @valhallasw I added a legend. [15:40:26] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Kubernetes worker nodes hanging - https://phabricator.wikimedia.org/T141017#2509690 (10yuvipanda) a:03yuvipanda [15:40:37] 06Labs, 10Tool-Labs: Puppet disabled on tools-worker-10* - https://phabricator.wikimedia.org/T141719#2509691 (10yuvipanda) See https://phabricator.wikimedia.org/T141017#2507131 [15:47:59] YuviPanda: puppet agent --disable "T12345" ;-) [15:48:06] T12345: Create "annotation" namespace on Hebrew Wikisource - https://phabricator.wikimedia.org/T12345 [16:21:28] valhallasw`cloud yeah, my mistake :| [16:37:10] YuviPanda: do you mean nfsiostat? [16:37:15] there is no iostat.io for tools-merlbot-proxy [16:37:36] hmm, iostat.vda / iostat.dm-0? [16:37:40] let me look [16:37:50] I should make a simple grafana dashboard for it [16:38:04] valhallasw`cloud grafana-labs-admin.wikimedia.org happeneed as well, btw [16:38:18] ah, .iostat.vda.io [16:38:44] not sure why searching for merlbot-proxy.*iostat didn't show that :/ [16:39:17] right [16:40:18] and how is 'io spikes' defined? [16:41:07] (above which value?) [16:41:46] 06Labs: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2509712 (10valhallasw) [16:44:10] valhallasw`cloud just looking 'spiky' no specific value [16:44:14] ok [16:51:23] 06Labs, 10Labs-Other-Projects, 10Tool-Labs, 10Graphite, 06WMDE-Analytics-Engineering: Update operations-software-grafana - https://phabricator.wikimedia.org/T141726#2509713 (10Addshore) [16:58:16] 06Labs, 10Labs-Other-Projects, 10Tool-Labs, 10Graphite, 06WMDE-Analytics-Engineering: Update operations-software-grafana - https://phabricator.wikimedia.org/T141726#2509728 (10yuvipanda) Are you sure our Grafana is 1.9? ``` yuvipanda@labmon1001:~$ apt-cache show grafana Package: grafana Priority: extra... [17:11:54] 06Labs, 10Labs-Other-Projects, 10Tool-Labs, 10Graphite, 06WMDE-Analytics-Engineering: Update operations-software-grafana - https://phabricator.wikimedia.org/T141726#2509738 (10Addshore) I was amusing grafana was deployed from operations-software-grafana ? *goes to check in puppet*.... [17:13:24] 06Labs, 10Labs-Other-Projects, 10Tool-Labs, 10Graphite, 06WMDE-Analytics-Engineering: Update operations-software-grafana - https://phabricator.wikimedia.org/T141726#2509739 (10Addshore) Ahh, I see no mention of that repo but I do see ``` package { 'grafana': ensure => present, } ``` Perha... [17:13:34] 06Labs, 10Labs-Other-Projects, 10Tool-Labs, 06WMDE-Analytics-Engineering: Add simple-json-datasource plugin to operations-software-grafana - https://phabricator.wikimedia.org/T141636#2509741 (10Addshore) [17:13:37] 06Labs, 10Labs-Other-Projects, 10Tool-Labs, 10Graphite, 06WMDE-Analytics-Engineering: Update operations-software-grafana - https://phabricator.wikimedia.org/T141726#2509740 (10Addshore) 05Open>03Invalid [17:14:37] YuviPanda: can you add [INACTIVE] or something to the start of the repo description? [17:15:49] infact, is there a place to request making them read only / marking as inactive? I'm guessing the answer is file a phab ticket... [17:17:53] probably, addshore [17:17:59] *goes to do that* [17:20:53] 06Labs, 10Labs-Other-Projects, 10Tool-Labs, 06WMDE-Analytics-Engineering: Add simple-json-datasource plugin to labs grafana instance - https://phabricator.wikimedia.org/T141636#2505596 (10Addshore) [17:41:31] YuviPanda: ^^ if you fancy a quick review to guide in in regards to adding a require? Can I do require class['::role::grafana::base'] or sometihng? [17:50:39] 06Labs, 10Labs-Other-Projects, 10Tool-Labs, 06WMDE-Analytics-Engineering, 15User-Addshore: Add simple-json-datasource plugin to labs grafana instance - https://phabricator.wikimedia.org/T141636#2505596 (10Addshore) [17:52:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Freddy2001 was created, changed by Freddy2001 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Freddy2001 edit summary: Created page with "{{Tools Access Request |Justification=do bot work |Completed=false |User Name=Freddy2001 }}" [18:32:02] PROBLEM - SSH on tools-merlbot-proxy is CRITICAL: Server answer [19:27:03] RECOVERY - SSH on tools-merlbot-proxy is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [19:33:01] PROBLEM - SSH on tools-merlbot-proxy is CRITICAL: Server answer [19:43:02] RECOVERY - SSH on tools-merlbot-proxy is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [20:22:43] 06Labs: confirm that new base labs base image is adequate for kubernetes &c. - https://phabricator.wikimedia.org/T134944#2510025 (10Andrew) a:05Andrew>03yuvipanda ok... I made a new image called "Debian-Jessie (testing)" which should be better. Check my work? [21:21:59] hey im having issues loading login.tools.wmflabs.org [21:28:35] Hello? [21:30:45] Zppix, loading it? [21:31:05] im dumb u dont use that link in a browser .... [21:31:08] wow ignore me [21:31:24] it's ssh not http [21:35:18] i found that out :P [21:36:32] whats a toolaccount [21:39:37] Zppix, it probably refers to the service groups/accounts used in the tools project [21:39:46] ok how do i take ownership [21:40:09] of an existing service? [21:43:43] yes [21:43:46] i created an tool [21:44:22] if you created it you should be good to go [21:45:15] k im going back to mw dev this is confusing rn [21:55:11] Zppix: to get into a tool's (service group) account use "become " (without the quotes) in an SSH session with login.tools.wmflabs.org [21:55:25] i did that [21:56:19] Zppix: so what do you want to do? [21:58:55] I figured my problem out im now taking a break from labs and working on MW [21:59:37] OK. [22:49:57] OK here's a question. Is there a web-based interface to access the databases? Like we used to have PhpMyAdmin on the Toolserver... [22:51:14] there's quarry [22:51:32] I know about Quarry. I was looking for more of a PMA experience. [22:53:25] See "[Labs-l] Suspended instances and phpmyadmin" [22:54:00] Okay. [22:56:45] Okay, guess is off to configure MySQL Workbench... again... [23:01:23] it has been 3-4 years so maybe things have changed but I wouldn't install something like PMA without running it by labs ops [23:02:12] Of course.