[04:26:22] ty tgr|away :) [04:29:16] !log ores restarted uwsgi-ores service on ores-web-03 [04:29:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [04:31:23] !log ores manually killed since "python" process consuming 80% of memory and restarted uwsgi-ores service on ores-web-03 [04:31:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [06:40:05] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Not Available - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 531 bytes in 0.028 second response time [06:45:04] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.148 second response time [06:55:37] PROBLEM - Puppet run on tools-docker-registry-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:07:48] 10PAWS: python3-tk package missing - https://phabricator.wikimedia.org/T145362#2627564 (10Tbayer) [07:35:38] RECOVERY - Puppet run on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:59:25] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [09:48:09] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [09:57:28] 06Labs: Request creation of WikiDocumentaries labs project - https://phabricator.wikimedia.org/T145369#2627871 (10Susannaanas) [11:04:42] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [12:14:43] 06Labs, 07LDAP, 13Patch-For-Review: Restore ldaplist -l passwd - https://phabricator.wikimedia.org/T122595#1908530 (10hashar) Note that T114063 is about dropping `ldapsupportlib.py` and thus `ldaplist`. [13:33:46] (03PS1) 10Andrew Bogott: Add labspuppetbackend::mysql_password: dummy for the puppet compiler [labs/private] - 10https://gerrit.wikimedia.org/r/309988 [13:34:10] (03PS2) 10Andrew Bogott: Add labspuppetbackend::mysql_password: dummy for the puppet compiler [labs/private] - 10https://gerrit.wikimedia.org/r/309988 [13:34:18] (03CR) 10Andrew Bogott: [C: 032 V: 032] Add labspuppetbackend::mysql_password: dummy for the puppet compiler [labs/private] - 10https://gerrit.wikimedia.org/r/309988 (owner: 10Andrew Bogott) [13:47:18] !log ores restarted precached service on ores-web-03 [13:47:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [13:58:15] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:04:38] RECOVERY - Puppet staleness on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:18:22] 06Labs, 10Tool-Labs: puppet disabled on tools-exec-1410 - https://phabricator.wikimedia.org/T145274#2628719 (10chasemp) 05Open>03Resolved a:03chasemp Resolved :) [15:21:40] 06Labs, 10Tool-Labs: toollabs-webservice : Depends: python (>= 2.7.5-5~) but 2.7.3-0ubuntu2.2 is to be installed - https://phabricator.wikimedia.org/T145326#2628938 (10bd808) It's related but I think it goes back further in the history to {rOSTW06e1aa8}. Apparently the dh-python package which provides support... [15:43:35] 10Wikibugs, 06Multimedia, 10grrrit-wm: New Phabricator tasks and new Gerrit changesets about FileAnnotations should be reported in #wikimedia-multimedia IRC channel - https://phabricator.wikimedia.org/T144997#2629044 (10Aklapper) [15:45:37] (03Draft1) 10Paladox: Add mediawiki/extensions/FileAnnotations to #wikimedia-multimedia channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310027 (https://phabricator.wikimedia.org/T144997) [15:46:33] (03PS2) 10Paladox: Add mediawiki/extensions/FileAnnotations to #wikimedia-multimedia channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310027 (https://phabricator.wikimedia.org/T144997) [15:47:27] (03Draft1) 10Paladox: Add FileAnnotations to #wikimedia-multimedia channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/310028 (https://phabricator.wikimedia.org/T144997) [15:48:12] (03PS2) 10Paladox: Add FileAnnotations to #wikimedia-multimedia channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/310028 (https://phabricator.wikimedia.org/T144997) [15:50:56] 10Wikibugs, 06Multimedia, 10grrrit-wm, 13Patch-For-Review: New Phabricator tasks and new Gerrit changesets about FileAnnotations should be reported in #wikimedia-multimedia IRC channel - https://phabricator.wikimedia.org/T144997#2616436 (10Paladox) I can deploy this one https://gerrit.wikimedia.org/r/31002... [15:54:18] 06Labs: Request increased quota for reading-web-staging labs project - https://phabricator.wikimedia.org/T144623#2605517 (10chasemp) It seems like there was some cleanup here that rendered the quota bump unnecessary. Can I get confirmation on where this is at? If still needed can this need be phrased in terms... [15:55:46] 06Labs: Request creation of labs-vpn labs project - https://phabricator.wikimedia.org/T143939#2629093 (10chasemp) 05Open>03stalled Please make a task for the discussion of https://wikitech.wikimedia.org/wiki/Labs_VPN_proposal and then when it's agreed we can move forward this ticket would be applicable. We'... [15:55:48] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2629095 (10chasemp) [15:57:26] !log wikilabels deployed wikilabels-wmflabs-deploy:578ec68 [15:57:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master [15:58:37] 06Labs: Request creation of wmwcourse labs project - https://phabricator.wikimedia.org/T144388#2598041 (10chasemp) Summation here is: * Quota for 20 small instances that will run mediawiki-vagrant * Will be removable in January 2017 (seems like it's for a course that runs through Dec) [16:06:19] 06Labs, 10Tool-Labs: toollabs-webservice : Depends: python (>= 2.7.5-5~) but 2.7.3-0ubuntu2.2 is to be installed - https://phabricator.wikimedia.org/T145326#2629158 (10bd808) @yuvipanda suggested building without pbuilder on tools-bastion-02. I did that, but still seem to get a package with the same dependency... [16:14:16] yuvipanda hi could you merge https://gerrit.wikimedia.org/r/#/c/308949/ and https://gerrit.wikimedia.org/r/#/c/310027/ please? [16:14:38] Would you also be able to have a look at https://gerrit.wikimedia.org/r/#/c/309307/ please? [16:16:36] paladox I'll have no time at all this week, sorry [16:16:56] Ok, yuvipanda i can deploy it, i just need someone to +2 [16:16:57] please [16:17:08] no, sorry. [16:17:12] Ok [16:17:54] (03CR) 10Bartosz Dziewoński: "Hmm, this might no longer be necessary after Andre adjusted Herald rule H13 (T144997#2619423)? I didn't realize that this just looks for t" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/310028 (https://phabricator.wikimedia.org/T144997) (owner: 10Paladox) [16:18:14] 10PAWS: python3-tk package missing - https://phabricator.wikimedia.org/T145362#2627564 (10yuvipanda) Have you tried using `%matplotlib inline` instead of `%matplotlib`? The former works better in notebooks. [16:18:30] (03CR) 10Bartosz Dziewoński: [C: 031] Add mediawiki/extensions/FileAnnotations to #wikimedia-multimedia channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310027 (https://phabricator.wikimedia.org/T144997) (owner: 10Paladox) [16:23:51] (03Draft2) 10Paladox: Update some packages [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310033 [16:23:57] (03Draft1) 10Paladox: Update some packages [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310033 [16:24:10] (03CR) 10Paladox: "I have deployed this and it seems to be working :)" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310033 (owner: 10Paladox) [16:25:00] (03CR) 10Paladox: "Ive deployed this :)" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310027 (https://phabricator.wikimedia.org/T144997) (owner: 10Paladox) [16:25:17] !log tools.lolrrit-wm cherry picked https://gerrit.wikimedia.org/r/#/c/310027/ [16:25:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master [16:27:24] 10Wikibugs, 06Multimedia, 10grrrit-wm, 13Patch-For-Review: New Phabricator tasks and new Gerrit changesets about FileAnnotations should be reported in #wikimedia-multimedia IRC channel - https://phabricator.wikimedia.org/T144997#2629297 (10Paladox) I've deployed https://gerrit.wikimedia.org/r/#/c/310027/ n... [16:28:51] 06Labs, 10Labs-Infrastructure, 06Operations, 10Traffic, and 2 others: update *.wmflabs.org certificate (existing expires on 2016-09-16) - https://phabricator.wikimedia.org/T145120#2629309 (10chasemp) a:05chasemp>03yuvipanda [16:32:22] 06Labs: Request creation of WikiDocumentaries labs project - https://phabricator.wikimedia.org/T145369#2627871 (10Andrew) approved [16:35:05] paladox I've just removed myself as a maintainer on grrrit-wm and emailed wikitech-l about it. You'll need to find someone else to merge your changes now. sorry, and good luck! [16:35:17] Ok [16:35:34] paladox: good luck! [16:35:43] Ok :) [16:35:44] thanks [16:36:26] !log tools.lolrrit-wm remove self as maintainer, have no time nor bandwidth [16:36:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master [16:37:54] yuvipanda Hi, would i be able to request +2 rights on that repo? [16:38:03] Im not sure where the page is for doing that? [16:38:05] please [16:38:08] paladox: I don't know either :) [16:38:13] Oh [16:38:19] 06Labs, 07Tracking: New Labs project requests (tracking) - https://phabricator.wikimedia.org/T76375#2629360 (10chasemp) [16:38:21] 06Labs: Request creation of WikiDocumentaries labs project - https://phabricator.wikimedia.org/T145369#2629357 (10chasemp) 05Open>03Resolved a:03chasemp seconded and created no caps are allowed so 'wikidocumentaries' and you are both admins [16:38:32] Oh i think it might be https://phabricator.wikimedia.org/project/view/1076/ [16:42:16] yuvipanda https://phabricator.wikimedia.org/T145416 :), wonder if you could quickly -1 or +1 [16:42:20] please [16:42:20] ? [16:42:26] no [16:42:30] Ok [18:14:46] (03CR) 10Merlijn van Deen: [C: 032] Add mediawiki/extensions/FileAnnotations to #wikimedia-multimedia channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310027 (https://phabricator.wikimedia.org/T144997) (owner: 10Paladox) [18:15:11] (03CR) 10Paladox: "Thanks." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310027 (https://phabricator.wikimedia.org/T144997) (owner: 10Paladox) [18:15:18] (03Merged) 10jenkins-bot: Add mediawiki/extensions/FileAnnotations to #wikimedia-multimedia channel [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310027 (https://phabricator.wikimedia.org/T144997) (owner: 10Paladox) [18:15:38] (03CR) 10Merlijn van Deen: [C: 032] Add FileAnnotations to #wikimedia-multimedia channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/310028 (https://phabricator.wikimedia.org/T144997) (owner: 10Paladox) [18:16:17] (03CR) 10Paladox: "This has been tested and seems to improve reliability but I carnt really confirm this but the bot seems to be connecting longer instead of" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310033 (owner: 10Paladox) [18:16:21] (03CR) 10Paladox: [C: 031] Update some packages [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310033 (owner: 10Paladox) [18:16:54] (03Merged) 10jenkins-bot: Add FileAnnotations to #wikimedia-multimedia channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/310028 (https://phabricator.wikimedia.org/T144997) (owner: 10Paladox) [18:17:36] (03CR) 10Merlijn van Deen: [C: 032] Update some packages [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310033 (owner: 10Paladox) [18:18:14] (03Merged) 10jenkins-bot: Update some packages [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310033 (owner: 10Paladox) [18:19:45] (03CR) 10Paladox: "Thanks." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/310033 (owner: 10Paladox) [18:23:46] (03PS5) 10Paladox: Do not show merges by the L10n-bot also update packages [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/308949 (https://phabricator.wikimedia.org/T93082) [18:25:18] (03PS6) 10Paladox: Do not show merges by the L10n-bot [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/308949 (https://phabricator.wikimedia.org/T93082) [18:25:44] (03CR) 10Paladox: [C: 031] "Was deployed and didn't distrupt anything else. I carnt confirm this fixes it but it should." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/308949 (https://phabricator.wikimedia.org/T93082) (owner: 10Paladox) [18:26:03] !log tools Built toollabs-webservice v0.27 package and added to aptly [18:26:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:26:24] !log tools Forcing puppet run on tools-precise-dev [18:26:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:28:43] !log tools Forcing puppet run on tools-bastion-02 [18:28:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:31:34] !log tools Forcing puppet run on tools-bastion-03 [18:31:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:33:16] !log tools Forcing puppet run on tools-cron-01 [18:33:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [18:37:15] RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [18:37:42] 06Labs, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: toollabs-webservice : Depends: python (>= 2.7.5-5~) but 2.7.3-0ubuntu2.2 is to be installed - https://phabricator.wikimedia.org/T145326#2629919 (10bd808) 05Open>03Resolved a:03bd808 Explicitly setting the python version deps in debian/control see... [18:44:07] 06Labs: Request creation of wmwcourse labs project - https://phabricator.wikimedia.org/T144388#2629978 (10Nikerabbit) Yep. I said early 2017 in case the project work takes more time to finish after the lectures end in December. [18:49:22] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:43] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:58:19] RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [18:58:51] RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [18:59:23] RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [19:02:13] RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [19:03:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [19:03:48] RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:10] RECOVERY - Puppet run on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:36] RECOVERY - Puppet run on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [19:17:43] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:18:54] 06Labs, 10Labs-Infrastructure, 06Operations, 10Traffic, and 2 others: update *.wmflabs.org certificate (existing expires on 2016-09-16) - https://phabricator.wikimedia.org/T145120#2630146 (10yuvipanda) I've moved over tools-static and tools, and they're good now. Need to do novaproxy next. [19:27:28] 06Labs, 10Labs-Infrastructure, 06Operations, 10Traffic, and 2 others: update *.wmflabs.org certificate (existing expires on 2016-09-16) - https://phabricator.wikimedia.org/T145120#2630182 (10yuvipanda) I had to do this for the following set of hosts on tools: 1. tools-proxy-* 2. tools-static-* I've done... [20:06:05] 10Wikibugs, 06Multimedia, 10grrrit-wm, 13Patch-For-Review: New Phabricator tasks and new Gerrit changesets about FileAnnotations should be reported in #wikimedia-multimedia IRC channel - https://phabricator.wikimedia.org/T144997#2630336 (10matmarex) 05Open>03Resolved Thanks folks! [20:07:27] (03CR) 10Paladox: "Thanks." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/310028 (https://phabricator.wikimedia.org/T144997) (owner: 10Paladox) [20:20:50] PROBLEM - Puppet run on tools-proxy-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:27:49] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Joeygreen1 was modified, changed by Joeygreen1 link https://wikitech.wikimedia.org/w/index.php?diff=829290 edit summary: [20:32:14] hello. i'm actually trying to retrieve some results from database tables and it gives me the message: command denied to user. is there any general problem or it could be related to my login? [20:34:13] horizon throws errors? [20:34:33] nvm, works now [20:35:04] http://pastebin.com/zhUKzCXb [20:35:49] RECOVERY - Puppet run on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:36:55] marmick: hmm... seems to be some issue with the user credentials you are using. I can run the same query using my user account's crednetials. [20:37:10] ok bd808... what can I do? [20:38:37] marmick: I'm looking in phabricator. I think we have a tracking bug for issues like this... [20:39:21] bd808: Wasn't it related to credentials being created wrong/incomplete a while back? I remember something about that... [20:39:53] yeah I think that's what I was remembering. I found T135931 [20:39:54] T135931: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931 [20:40:03] which is similar but different [20:41:39] marmick: literally the easiest thing to do would be to create a new tool and copy the replica.my.cnf from it. I'm not sure that's the most recommended solution however [20:42:11] bd808: Could you run the grants on that account? [20:42:25] bd808: i checked my account root directory and i found the replica [20:42:28] .my.cnf [20:42:38] the error was given when i tried using the terminal [20:42:53] i don't know if it would fail connecting through a python script, i haven't tried it yet [20:43:27] hi [20:43:35] oh, so you do have a .my.cnf [20:43:40] * Matthew_ waves to yuvipanda :) [20:44:52] Matthew_: I can't show that user's grants (not enough rights) but marmick could try running "show grants;" theirself [20:45:39] Okay. Maybe you could fix it by granting any missing permissions? [20:45:40] we would expect a grant like "GRANT SELECT, SHOW VIEW ON `%\_p`.* TO 'u3532'@'%'" if everything is normal [20:48:02] marmick: i think the problem is you're using cawikibooks instead of cawikibooks_p [20:48:09] yuvipanda: that's it! [20:48:12] i just tried it now [20:48:21] i thought it was the same [20:48:22] yeah. we just figured that out in a PM [20:48:24] :) [20:48:27] but apparently it isn't! [20:48:31] :) [20:48:35] thanks [20:48:48] nice [20:55:53] 06Labs, 10Labs-Infrastructure, 06Operations, 10Traffic, and 2 others: update *.wmflabs.org certificate (existing expires on 2016-09-16) - https://phabricator.wikimedia.org/T145120#2630676 (10yuvipanda) ok, done on novaproxy-01 and -02 as well! I've also documened the tools ssl certs in https://wikitech.w... [21:04:17] 06Labs, 10Labs-Infrastructure, 06Operations, 10Traffic, and 2 others: update *.wmflabs.org certificate (existing expires on 2016-09-16) - https://phabricator.wikimedia.org/T145120#2630718 (10yuvipanda) 05Open>03Resolved [21:04:42] (03PS2) 10Lokal Profil: [NOT READY] Store plain permalink instead of urlencoded one [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309858 (https://phabricator.wikimedia.org/T112460) [21:26:04] (03PS3) 10Lokal Profil: [NOT READY] Store plain permalink instead of urlencoded one [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309858 (https://phabricator.wikimedia.org/T112460) [21:31:04] (03PS1) 10Lokal Profil: Correctly indent one bit in api/FormatCsv [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/310142 [21:33:51] o/ [21:33:57] I'm struggling to log into horizon. [21:34:13] I've copy-pasted the same creds from wikitech and still get "Invalid credentials" [21:34:56] Oh weird. My two factor auth got disabled. [21:36:26] 2fa was disabled on your account and you don't know about it? [21:37:32] I think I might know what happened. [21:38:23] (03CR) 10Jean-Frédéric: "Hmmm, I do agree with your comment in 309842 ; however I meant this test to ensure that before and after the refactoring would be the same" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309841 (owner: 10Jean-Frédéric) [21:38:54] (03CR) 10Jean-Frédéric: Extract CommonsCat templates data to JSON file (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309842 (owner: 10Jean-Frédéric) [21:40:01] I disabled and re-enabled my 2_factor and I think I might have not validated when I re-enabled. [22:22:23] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.79 ms [22:32:20] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [22:36:12] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.51 ms [22:37:18] 10Labs-project-other, 10Quarry, 06Discovery, 10Wikidata, and 2 others: Setup sparqly service at https://sparqly.wmflabs.org/ (like Quarry) - https://phabricator.wikimedia.org/T104762#2631095 (10Smalyshev) p:05Triage>03Low [22:38:08] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [22:43:39] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.81 ms [23:02:26] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [23:03:43] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:13:39] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0]