[00:39:08] <wikibugs>	 06Labs, 10MediaWiki-extensions-OpenStackManager: WikiPage::something error encountered while adding two users to Tools project at the same time - https://phabricator.wikimedia.org/T133742#2262268 (10scfc) p:05Triage>03Lowest T91988 would make this report probably obsolete, yes.  I personally don't close bu...
[03:13:50] <bd808>	 !log tools.hatjitsu Joined project to try and fix it up
[04:21:12] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:28:43] <bd808>	 !log tools.hatjitsu It's alive!
[04:28:48] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.hatjitsu/SAL, Master
[04:35:21] <wikibugs>	 10Tool-Labs: Install xml2 on labs - https://phabricator.wikimedia.org/T134146#2262473 (10Kelson) @scfc For both
[04:36:32] <wikibugs>	 06Labs, 06Team-Practices, 07Privacy: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2262474 (10bd808) I have a modified version of Hatjitsu up and running on Tool Labs at https://tools.wmflabs.org/hatjitsu/. This is a refresh of a tool th...
[07:03:14] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:08:08] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 824668 bytes in 3.524 second response time
[08:02:53] <grrrit-wm>	 (03CR) 10Lokal Profil: [C: 032] "Merging this now. I'll do a follow-up to address the remaining details of T132647 and the i18n message." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil)
[08:03:37] <grrrit-wm>	 (03Merged) 10jenkins-bot: [NOT FULLY TESTED] Better support for sister projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286156 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil)
[08:34:52] <shinken-wm>	 PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228)
[08:35:44] <grrrit-wm>	 (03PS1) 10Lokal Profil: Change i18n message to allow non-Wikipedia projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286796 
[09:02:15] <grrrit-wm>	 (03PS1) 10Lokal Profil: Support non-Wikipedia projects in pretty urls [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286798 (https://phabricator.wikimedia.org/T132647) 
[09:04:58] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Upgrade db1069 - https://phabricator.wikimedia.org/T134349#2262819 (10jcrespo)
[09:15:01] <grrrit-wm>	 (03PS1) 10Lokal Profil: Invert order of assert parameters (PHP) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286800 
[09:18:09] <grrrit-wm>	 (03CR) 10Lokal Profil: "This makes the test output a bit less confusing =)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286800 (owner: 10Lokal Profil)
[09:26:29] <grrrit-wm>	 (03CR) 10Lokal Profil: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286149 (https://phabricator.wikimedia.org/T39974) (owner: 10Lokal Profil)
[09:30:07] <grrrit-wm>	 (03CR) 10Lokal Profil: "As this change is fairly simple and there isn't a straight forward way of testing it I would suggest reviewing + merge/deploy as is." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286149 (https://phabricator.wikimedia.org/T39974) (owner: 10Lokal Profil)
[09:50:19] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA: Lost database changes on s2 for 3 hours on labs replicas - https://phabricator.wikimedia.org/T129432#2263040 (10jcrespo)
[09:50:25] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: labsdb1001 short on available space - https://phabricator.wikimedia.org/T132431#2263041 (10jcrespo)
[09:50:34] <wikibugs>	 06Labs, 10DBA, 06Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2263042 (10jcrespo)
[09:50:44] <wikibugs>	 06Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: watchlist table not available on labs - https://phabricator.wikimedia.org/T59617#2263045 (10jcrespo)
[09:50:48] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA, 13Patch-For-Review: Tool Labs queries die - https://phabricator.wikimedia.org/T127266#2263046 (10jcrespo)
[09:50:52] <wikibugs>	 06Labs, 10DBA: Data missing from June 11/12 on s3.labsdb - https://phabricator.wikimedia.org/T115517#2263047 (10jcrespo)
[09:53:36] <grrrit-wm>	 (03CR) 10Aklapper: "@Legoktm: A gentle ping to consider deploying this (as it blocks the other eight-line patch). :)" [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/240422 (owner: 10Legoktm)
[10:42:06] <Krinkle>	 !log morebots cvn-morebots is gone
[10:42:06] <stashbot>	 Did you mean tools.morebots instead of morebots?
[10:42:07] <labs-morebots>	 morebots is not a valid project.
[10:42:15] <Krinkle>	 !log tools.morebots cvn-morebots is gone
[10:42:19] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.morebots/SAL, Master
[10:47:38] <Krinkle>	 !log tools.morebots Started cvn-morebots job
[10:47:42] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.morebots/SAL, Master
[10:52:28] <grrrit-wm>	 (03PS2) 10Krinkle: Change i18n message to allow non-Wikipedia projects [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286796 (owner: 10Lokal Profil)
[10:55:51] <grrrit-wm>	 (03CR) 10Krinkle: "Regarding the order of merge/deploy, I recommend migrating to using Intuition as a library instead of Intuition as embedded installation. " [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286796 (owner: 10Lokal Profil)
[11:05:25] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] Support non-Wikipedia projects in pretty urls [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286798 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil)
[11:06:06] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 032] "Indeed :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286800 (owner: 10Lokal Profil)
[11:07:06] <grrrit-wm>	 (03Merged) 10jenkins-bot: Support non-Wikipedia projects in pretty urls [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286798 (https://phabricator.wikimedia.org/T132647) (owner: 10Lokal Profil)
[11:07:55] <grrrit-wm>	 (03CR) 10Jean-Frédéric: [C: 031] [NOT TESTED] Remove monument_random from database and api. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286149 (https://phabricator.wikimedia.org/T39974) (owner: 10Lokal Profil)
[11:08:20] <grrrit-wm>	 (03Merged) 10jenkins-bot: Invert order of assert parameters (PHP) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/286800 (owner: 10Lokal Profil)
[11:11:01] <wikibugs>	 06Labs: MW database: user.user_editcount shows a wrong value - https://phabricator.wikimedia.org/T134359#2263376 (10Danny_B)
[11:27:58] <wikibugs>	 06Labs: MW database: user.user_editcount shows a wrong value - https://phabricator.wikimedia.org/T134359#2263175 (10jcrespo) This is not a replica issue- it mirrors its value on production.  I would assume one of these: a) the field is not used b) it is updated by a maintenance script that is not executed (or it...
[11:30:29] <Urbanecm_>	 Hi, when I try to login into Tool Labs from another SSH server (I tried two), it throws
[11:30:31] <Urbanecm_>	 -bash: warning: setlocale: LC_ALL: cannot change locale (cs_CZ.UTF-8)
[11:30:33] <Urbanecm_>	 -bash: warning: setlocale: LC_ALL: cannot change locale (cs_CZ.UTF-8)
[11:30:35] <Urbanecm_>	 -bash: warning: setlocale: LC_ALL: cannot change locale (cs_CZ.UTF-8)
[11:30:37] <Urbanecm_>	 And I am not able to write Czech chars. 
[11:30:39] <Urbanecm_>	 I tried to login from labs bastion host, no warning. 
[11:30:41] <Urbanecm_>	 Does anybody know why this happen? Thanks for reply. 
[11:32:12] <wikibugs>	 06Labs, 10DBA: MW database: user.user_editcount shows a wrong value - https://phabricator.wikimedia.org/T134359#2263410 (10Aklapper)
[11:49:46] <grrrit-wm>	 (03PS99) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 
[11:54:30] <grrrit-wm>	 (03CR) 10Ricordisamoa: "PS99 adds DraggableElement.prototype.clone and EditableDraggableElement.prototype.clone, uses them instead of the jQuery .clone() to fix '" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa)
[12:15:32] <grrrit-wm>	 (03PS100) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 
[12:20:59] <grrrit-wm>	 (03CR) 10Ricordisamoa: "PS100 renames the 'draggable' param in Section.prototype.getDraggable and LanguageSection.prototype.getDraggable to 'config'" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa)
[12:21:39] <grrrit-wm>	 (03PS101) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 
[12:26:53] <grrrit-wm>	 (03CR) 10Ricordisamoa: "PS101 moves the SimpleTermFormatter class definition up" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa)
[12:52:07] <wikibugs>	 06Labs, 10DBA: MW database: user.user_editcount shows a wrong value - https://phabricator.wikimedia.org/T134359#2263688 (10jcrespo) @Aklapper preciselly, this is not a DBA issue (nothing is wrong with the database itself). I think it is an Invalid one, but I do not know too much about mediawiki to be 100% sure.
[12:52:18] <wikibugs>	 06Labs, 10DBA: MW database: user.user_editcount shows a wrong value - https://phabricator.wikimedia.org/T134359#2263689 (10jcrespo)
[12:57:18] <grrrit-wm>	 (03PS102) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 
[12:58:41] <Glaisher>	 o_O PS102
[13:00:30] <wikibugs>	 06Labs, 10DBA, 10MediaWiki-General-or-Unknown: MW database: user.user_editcount shows a wrong value - https://phabricator.wikimedia.org/T134359#2263743 (10Glaisher)
[13:01:10] <grrrit-wm>	 (03CR) 10Ricordisamoa: "PS102 makes terms contenteditable in LanguageSection.prototype.getDraggable instead of SimpleTermFormatter" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa)
[13:04:57] <grrrit-wm>	 (03CR) 10Ricordisamoa: "Example response length on real data:" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa)
[13:38:59] <grrrit-wm>	 (03PS1) 10Elukey: Add fake database password for hue [labs/private] - 10https://gerrit.wikimedia.org/r/286844 (https://phabricator.wikimedia.org/T127990) 
[13:39:23] <grrrit-wm>	 (03CR) 10Elukey: [C: 032 V: 032] Add fake database password for hue [labs/private] - 10https://gerrit.wikimedia.org/r/286844 (https://phabricator.wikimedia.org/T127990) (owner: 10Elukey)
[13:44:24] <wikibugs>	 06Labs, 10Tool-Labs: Update nginx on tools and labs proxies and static file server - https://phabricator.wikimedia.org/T134383#2263935 (10yuvipanda)
[14:34:07] <Niharika>	 o/ I have a db (with a table) for a tool I'm a co-maintainer of. I get 'UPDATE command denied to user 's52615'@'10.68.23.58' for table 'copyright_diffs'' when I try to modify the table. Any ideas what am I missing here?
[14:37:22] <Niharika>	 YuviPanda: ^ if you're around!
[14:51:35] <bd808>	 Niharika: which database? I'm not finding anything on tools-db that is obviously owned by s52615.
[14:52:28] <Niharika>	 bd808: "s51306__copyright_p" which is clearly owned by s51306. I think I'll have to ask eranroz about this. 
[14:53:34] <Niharika>	 bd808: Can two tools share a database/table?
[14:53:37] <jynus>	 I can add shared edit rights
[14:53:46] <jynus>	 but obviously I need permission from the owner
[14:53:57] <Niharika>	 jynus: Email will do?
[14:54:06] <jynus>	 better, ticket or it will get lost
[14:54:06] <bd808>	 Niharika: yeah, all the *_p tables are public
[14:54:09] <jynus>	 I can do it in seconds
[14:54:21] <Niharika>	 jynus: Okay, will add one. 
[14:54:58] <bd808>	 Niharika: 51306 is tools.eranbot
[14:54:59] <jynus>	 clarify as much as possible (owner, permissions wanted, etc.)
[14:55:14] <bd808>	 you can find that sort of thing out using https://tools.wmflabs.org/contact/
[14:55:31] <Niharika>	 bd808: Aha, okay. Thanks!
[14:55:49] <Niharika>	 jynus: On it. Which project should I associate the ticket to?
[14:56:04] <jynus>	 DBA + labs infrastructure
[14:56:13] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1203 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[14:57:53] <halfak>	 !log wikilabels SQL on prod DB: CREATE INDEX workset_user_campaign ON workset (user_id, campaign_id);
[14:57:57] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL, Master
[14:58:05] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[14:59:30] <halfak>	 Taking a long time to drop workset_user. 
[15:01:41] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[15:05:38] <shinken-wm>	 PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[15:08:32] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[15:10:36] <shinken-wm>	 PROBLEM - Puppet run on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[15:11:01] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA: Add shared edit rights between eranbot and plagiabot tools - https://phabricator.wikimedia.org/T134392#2264219 (10Niharika)
[15:11:34] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA: Add shared edit rights between eranbot and plagiabot tools - https://phabricator.wikimedia.org/T134392#2264235 (10Niharika) @eranroz, this needs your permission to proceed. :) Thanks!
[15:13:33] <halfak>	 Uhoh
[15:13:45] <halfak>	 One of my VMs just went down. 
[15:14:09] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[15:15:25] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[15:16:35] <shinken-wm>	 PROBLEM - Puppet run on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[15:16:55] <shinken-wm>	 PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[15:17:01] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]
[15:17:11] <shinken-wm>	 PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[15:17:38] <halfak>	 Looks like lots of things are down?
[15:17:41] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[15:18:11] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[15:18:41] <halfak>	 and we're back
[15:20:25] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[15:20:54] <shinken-wm>	 PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[15:21:54] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[15:32:25] <YuviPanda>	 halfak: temp dns blip
[15:32:36] <halfak>	 Gotcha.  Thanks YuviPanda 
[15:32:57] <halfak>	 was it big enough to warrant a write-up?
[15:33:14] <halfak>	 If not, I'll just post "temp DNS blib" to my ORES users and call it good.
[15:33:39] <YuviPanda>	 halfak: probably not, since we're doing a bunch of DNS stuff in the next few days to shore up DNS
[15:34:21] <halfak>	 kk
[15:36:13] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:38:05] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:41:37] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:45:36] <shinken-wm>	 RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:50:44] <shinken-wm>	 RECOVERY - Puppet run on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:52:02] <shinken-wm>	 RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:56:36] <shinken-wm>	 RECOVERY - Puppet run on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:56:50] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:57:06] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:57:38] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:03:10] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:05:35] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:07:15] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:12:07] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 824604 bytes in 3.904 second response time
[16:14:59] <tom29739>	 What's the point of having that check on the tool labs home page?
[16:15:25] <wikibugs>	 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#2264606 (10yuvipanda) I've killed it again just now, since it was killing the other webservices on the  webgrid node it was on.  If nobody else steps up to maintain / fix the issues with the application, I'll still try to brin...
[16:15:27] <tom29739>	 It seems to be very unreliable. And not indicative of tool labs going down.
[16:17:16] <bd808>	 tom29739: yeah. c.hase is planning on finding some better way to monitor the web grid.
[16:17:29] <bd808>	 monitoring is hard :/
[16:17:59] <YuviPanda>	 it did tell us 'kmlexport  is killing things again' so I've killed kmlexport and updated that bug
[16:22:36] <tom29739>	 Why don't you just monitor the webservice processes on the nodes?
[16:22:48] <tom29739>	 Or analyse the logs.
[16:23:03] <valhallasw`cloud>	 because those options are not 'just'
[16:23:08] <valhallasw`cloud>	 they take effort to implement
[16:23:32] <valhallasw`cloud>	 whereas the tool labs frontpage check is a trivial GET that is supported by the service we are using to monitor uptime
[16:24:06] <YuviPanda>	 we do indeed monitor and restart the webservice process, but that still will cause this test to flap anyway.
[16:24:26] <YuviPanda>	 and this tests multiple things - the proxy, the tool itself, LDAP, DNS, labsdb1004... so definitely not the best of tests :)
[16:24:28] <YuviPanda>	 brb
[16:29:49] <tom29739>	 How does production do it? I'd think they have some sort of solution to track for downtime...
[16:31:33] <wikibugs>	 10Tool-Labs: Install xml2 on labs - https://phabricator.wikimedia.org/T134146#2264640 (10scfc) p:05Triage>03High a:03scfc
[16:31:50] <valhallasw`cloud>	 tom29739: I'm not sure what your question is.
[16:31:56] <valhallasw`cloud>	 how does production do /what/?
[16:32:16] <tom29739>	 Monitor for downtime.
[16:32:21] <valhallasw`cloud>	 downtime of /what/
[16:33:03] <wikibugs>	 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2264647 (10scfc) Happened again today ~ 15:00Zish.
[16:33:05] <tom29739>	 Webservices on tools.
[16:33:22] <valhallasw`cloud>	 again, /what/ is it you want to measure
[16:33:40] <valhallasw`cloud>	 for 'is the tool labs web page online and responsive', our current check does exactly that
[16:34:43] <valhallasw`cloud>	 the problem with monitoring generally isn't so much the technical aspect, the issue is defining what you want to measure
[16:37:32] * valhallasw`cloud is gone for a bit
[16:42:22] <tom29739>	 Maybe monitor for 503, or 5xx responses from tools.
[16:47:41] <wikibugs>	 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1125310 (10bd808) The klmexport tool looks to be a [[https://en.wikipedia.org/wiki/Poster_child|poster child]] for the types of problems I was thinking of when I started describing [[https://meta.wikimedia.org/wiki/Community_T...
[16:52:56] <wikibugs>	 06Labs, 10Tool-Labs: kmlexport perl script memory usage - https://phabricator.wikimedia.org/T99236#2264736 (10bd808)
[16:52:58] <wikibugs>	 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#2264735 (10bd808)
[17:49:24] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:59:29] <wikibugs>	 10Labs-project-extdist: ExtensionDistributor Labs service down - https://phabricator.wikimedia.org/T134390#2265035 (10Legoktm) Works for me too...?  ``` km@km-tp ~> curl -I "https://extdist.wmflabs.org/dist/extensions/Flow-REL1_26-c89d781.tar.gz" HTTP/1.1 200 OK Server: nginx/1.9.4 Date: Wed, 04 May 2016 17:59:1...
[18:00:03] <Luke081515>	 !log rcm.cac updating to master
[18:00:08] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm.cac/SAL, Master
[18:00:12] <Luke081515>	 ah
[18:04:13] <wikibugs>	 06Labs, 06Team-Practices, 07Privacy: http://hatjitsu.wmflabs.org loads resources from numerous 3rd party sites - https://phabricator.wikimedia.org/T134288#2265057 (10KLans_WMF)
[18:30:13] <tom29739>	 Is there a way to get the host a job is running on, other than qstat?
[18:30:27] <valhallasw`cloud>	 No.
[18:30:39] <tom29739>	 thanks.
[18:36:19] <valhallasw`cloud>	 (it's a bit of a weird question to ask -- why would you want to use something else?)
[18:39:07] <tom29739>	 valhallasw`cloud, I'm trying to get it in a bash script, and I can get the job in qstat, and I have to filter it down to the specific host.
[18:39:31] <tom29739>	 I wondered whether there was something that gave it straight out.
[18:40:47] <valhallasw`cloud>	 no, you'll have to `qstat -xml` and parse that output
[18:40:56] <valhallasw`cloud>	 (`qstat -xml -j ...` doesn't return host information)
[18:41:43] <mutante>	 where does qstat get it from?
[18:42:00] <mutante>	 maybe it reads a file
[18:42:17] <valhallasw`cloud>	 dark SGE magic
[18:42:24] <mutante>	 strace qstat ? :p
[18:43:07] <tom29739>	 It's in the queue field in qstat, so it's maybe where it holds the queue info?
[18:43:15] <valhallasw`cloud>	 it connects to the SGE master process and sends an xml blob
[18:43:23] <valhallasw`cloud>	 aka dark SGE magic ;-)
[18:44:32] <valhallasw`cloud>	 tom29739: I'm not sure what you mean
[18:44:58] <valhallasw`cloud>	 the queue_name field is where you can get the hostname, but that field is not available if you pass -j <...> to specify a job
[18:46:07] <tom29739>	 why doesn't it show the host info when you do -j?
[18:46:45] <valhallasw`cloud>	 You'll have to ask Sun.
[18:48:32] <tom29739>	 When is tools moving to k8s? I've seen some phab tasks related to k8s on tools, and I'm not sure when it's actually happening.
[18:49:08] <valhallasw`cloud>	 the plan is to have k8s available for general usage somewhere in the second half of this year
[18:49:32] <YuviPanda>	 webservices within end of quarter (aka end of june)
[18:49:37] <YuviPanda>	 will have a --backend=k8s flag
[18:50:58] <tom29739>	 Will it change how I submit jobs, and stuff like that?
[18:51:33] <YuviPanda>	 for webservices? no, because we wrote and maintain the 'webservice' command
[18:52:07] <YuviPanda>	 for jsub? we don't know yet. We'll have stats on what commandline arguments get passed and we'll try to provide backwards compat for as many as possible, but some edge cases will break probably.
[18:52:21] <YuviPanda>	 but we aren't even going to start working on jsub for a while.
[18:56:15] <valhallasw`cloud>	 mutante: it connects, sends magic XML, then sends magic binary data
[18:56:41] <valhallasw`cloud>	 so unless you feel like reading '90s C, I would stay away from it ;-)
[18:56:56] <mutante>	 valhallasw`cloud: heh, no no.. we tried, it's ok :)
[19:13:41] <wikibugs>	 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#2265234 (10yuvipanda) To clarify, until some additional maintainers for this tool are found or we setup our kubernetes based container setup, this tool will stay down.
[19:13:48] <wikibugs>	 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#2265237 (10Para) With the [[https://www.mediawiki.org/wiki/Maps|Wikimedia Maps]] project looking promising, I'm not particularly interested in spending more time on the kmlexport tool. If someone wants to keep working on it, t...
[20:11:13] <wikibugs>	 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#2265474 (10bd808) >>! In T92963#2265237, @Para wrote: > With the [[https://www.mediawiki.org/wiki/Maps|Wikimedia Maps]] project looking promising, I'm not particularly interested in spending more time on the kmlexport tool.  T...
[21:19:01] <tom29739>	 tools-bastion-03 seems sluggish.
[21:32:00] <wikibugs>	 06Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2265794 (10chasemp) OK I caught the tail end of a brief storm this morning.  There indeed a query storm that hits the pdns recursor associated with an increase in servfail answers to clients within labs....