[00:51:18] 06Labs, 10Tool-Labs: Simplify and reduce the amount of options jsub supports - https://phabricator.wikimedia.org/T134846#2469699 (10Danny_B) [02:40:54] 10Tool-Labs-tools-Pageviews: The Pageviews lab tool shows the same link to "Most viewed pages" for different projects - https://phabricator.wikimedia.org/T140557#2468943 (10MusikAnimal) Fixed with [[ https://github.com/MusikAnimal/pageviews/commit/4b6e646cd090fc599f8fe9ec66f6e83c20f6e8e9 | 4b6e646 ]]. Thanks for... [02:41:07] 10Tool-Labs-tools-Pageviews: The Pageviews lab tool shows the same link to "Most viewed pages" for different projects - https://phabricator.wikimedia.org/T140557#2469744 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal [06:31:40] 06Labs, 10Tool-Labs, 10WLX-Jury, 13Patch-For-Review, 10Wiki-Loves-Monuments (2016): Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2469796 (10intracer) >>! In T124903#2333768, @yuvipanda wrote: > The only thing I'd need is that you provide me... [06:49:06] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Not Available - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 531 bytes in 0.015 second response time [06:59:05] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.029 second response time [07:02:41] 10PAWS: PAWS terminal doesn't support alt- on Mac OS X keyboard - https://phabricator.wikimedia.org/T139739#2469798 (10Mattias_Ostmar-WMSE) Nope, the problem is still there unfortunately. Started and stopped. [08:06:12] 06Labs, 10Labs-Infrastructure, 10Datasets-Archiving, 10Datasets-General-or-Unknown, 10Wikidata: [Bug] Wikidata JSON dumps gets deleted after every new Wikidata dump - https://phabricator.wikimedia.org/T107226#1490125 (10ArielGlenn) It seems that what would need to be changed is modules/dataset/files/labs... [08:20:40] 06Labs, 10Tool-Labs: /home/ilya missing replica.my.cnf - https://phabricator.wikimedia.org/T140592#2469834 (10intracer) [08:21:36] 06Labs, 10Tool-Labs: /home/ilya missing replica.my.cnf - https://phabricator.wikimedia.org/T140592#2469861 (10intracer) [08:21:38] 06Labs, 10Tool-Labs, 07Tracking: Tool Labs users missing replica.my.cnf (tracking) - https://phabricator.wikimedia.org/T135931#2469862 (10intracer) [08:23:21] 06Labs, 10Tool-Labs, 10WLX-Jury, 13Patch-For-Review, 10Wiki-Loves-Monuments (2016): Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2469878 (10intracer) [08:23:23] 06Labs, 10Tool-Labs: /home/ilya missing replica.my.cnf - https://phabricator.wikimedia.org/T140592#2469834 (10intracer) [08:23:40] 06Labs, 10Tool-Labs, 10WLX-Jury, 10Wiki-Loves-Monuments (2016): /home/ilya missing replica.my.cnf - https://phabricator.wikimedia.org/T140592#2469879 (10intracer) [09:45:05] 10PAWS: Paws display 502 - Bad gateway error - https://phabricator.wikimedia.org/T140578#2470014 (10OrenBochman) [11:17:09] 10Tool-Labs-tools-Pageviews: The Pageviews lab tool shows the same link to "Most viewed pages" for different projects - https://phabricator.wikimedia.org/T140557#2470194 (10Amire80) Confirmed fix, thanks! [12:03:20] 10PAWS: Paws display 502 - Bad gateway error - https://phabricator.wikimedia.org/T140578#2470253 (10yuvipanda) Can you try now? [12:04:47] 10PAWS: PAWS terminal doesn't support alt- on Mac OS X keyboard - https://phabricator.wikimedia.org/T139739#2470263 (10yuvipanda) Interesting. Can you type these into somewhere else, copy them and paste them into the terminal? [12:15:36] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Issues with 'webservice' kubernetes backend - https://phabricator.wikimedia.org/T139107#2470329 (10Magnus) There are still several of my tools that won't start kubernetes webservice because it thinks the gridengine one is still running. Sometimes it is (old gridengine... [12:30:07] (03PS1) 10Addshore: Add more projects to #wikimedia-de-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/299528 [12:40:33] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: etcd hosts hanging with kernel hang - https://phabricator.wikimedia.org/T140256#2470386 (10yuvipanda) 05Resolved>03Open Monitoring triggered, same node again! ``` tools-k8s-etcd-02 login: [278880.136697] INFO: task jbd2/vda3-8:109 blocked for more than 120 seconds... [12:41:03] !log tools reboot tools-k8s-etcd-02 for T140256 [12:41:04] T140256: etcd hosts hanging with kernel hang - https://phabricator.wikimedia.org/T140256 [12:41:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [12:48:19] !log tools reboot tools-flannel-etcd-03 for T140256 [12:48:19] T140256: etcd hosts hanging with kernel hang - https://phabricator.wikimedia.org/T140256 [12:48:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [12:55:02] 06Labs, 10Labs-Infrastructure: Add pp_propname/pp_value index to Labs replica - https://phabricator.wikimedia.org/T140609#2470422 (10Magnus) [12:58:44] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: etcd hosts hanging with kernel hang - https://phabricator.wikimedia.org/T140256#2470439 (10yuvipanda) So tools-k8s-etcd-02 came back up after reboot, but flannel-etcd-03 did not - it came back and kept getting hung with various other processes entering same state. I'm... [13:04:39] PROBLEM - Host tools-flannel-etcd-03 is DOWN: CRITICAL - Host Unreachable (10.68.22.169) [13:05:53] YuviPanda, ^ [13:06:02] yup, am migrating it atm [13:11:57] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: etcd hosts hanging with kernel hang - https://phabricator.wikimedia.org/T140256#2470480 (10yuvipanda) Ok, the cold migrate of tools-flannel-etcd-03 seems to have failed - I still can't ssh into it, and ```Current status is SHUTOFF; waiting for it to change to ACTIVE.... [13:13:41] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2470483 (10chasemp) [13:14:49] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2464178 (10chasemp) [13:16:00] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2470486 (10yuvipanda) Should get a new key setup for labs root, and probably good to rotate all keys anyway. [13:17:22] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2470490 (10yuvipanda) [13:18:16] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2470494 (10chasemp) [13:18:20] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2470495 (10yuvipanda) [13:19:36] 10PAWS: PAWS terminal doesn't support alt- on Mac OS X keyboard - https://phabricator.wikimedia.org/T139739#2470498 (10Mattias_Ostmar-WMSE) Copy pasting of the following chars to the terminal works fine, just not typing them in: {}[] [13:19:41] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2470499 (10chasemp) [13:21:06] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2464178 (10chasemp) [13:21:44] 10PAWS: PAWS terminal doesn't support alt- on Mac OS X keyboard - https://phabricator.wikimedia.org/T139739#2470505 (10yuvipanda) I wonder if this is caused by https://github.com/sourcelair/xterm.js/issues/124 [13:21:55] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2470506 (10chasemp) [13:23:05] 10PAWS: PAWS terminal doesn't support alt- on Mac OS X keyboard - https://phabricator.wikimedia.org/T139739#2470509 (10yuvipanda) If so, https://github.com/sourcelair/xterm.js/pull/175 should help fix it! [13:29:21] 10PAWS: PAWS terminal doesn't support alt- on Mac OS X keyboard - https://phabricator.wikimedia.org/T139739#2470525 (10Mattias_Ostmar-WMSE) Fingers crossed [13:39:40] RECOVERY - Host tools-flannel-etcd-03 is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms [14:44:52] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: etcd hosts hanging with kernel hang - https://phabricator.wikimedia.org/T140256#2458257 (10hashar) I had similar kernel lock on a single instance in May T134981 Since at least mid june Jenkins job had been stuck for ever being stall attempting to write to the virtual... [14:50:16] 06Labs, 10Labs-Infrastructure: Improvements to cold-migrate script - https://phabricator.wikimedia.org/T139272#2471119 (10Andrew) a:03Andrew [15:02:07] 06Labs, 10Labs-Infrastructure, 07Tracking: Labs instances sometimes freeze - https://phabricator.wikimedia.org/T124133#2471128 (10hashar) I havent heard of such stall Jenkins jobs last week. Would have to grep in the build history to be sure though by looking at job that took more than max timeout / got abor... [15:14:10] Labsadmin here? [15:14:39] in several back to back meetings here Luke081515 [15:14:50] I just need one job killed [15:15:26] AsuraBot is spamming about every 15 minutes at a talkpage. Killing job 8807036 (sum_disc) would help [15:15:42] !log tools kill 8807036 for Luke081515 [15:15:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [15:15:47] maybe the job would work correctly after restart [15:15:52] YuviPanda: thx :) [15:38:19] 10Tool-Labs-tools-Erwin's-tools: Unknown Error/MySQL errors - https://phabricator.wikimedia.org/T140421#2471257 (10Nemo_bis) >>! In T140421#2466261, @yuvipanda wrote: > 'webservice precise-lighttpd start' works now, and it is running in precise now. Not for me: ``` tools.erwin85@tools-bastion-03:~$ webservice... [15:38:47] 10Tool-Labs-tools-Erwin's-tools: Unknown Error/MySQL errors - https://phabricator.wikimedia.org/T140421#2471258 (10yuvipanda) Sorry, I meant 'lighttpd-precise' not precise-lighttpd. [15:53:58] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2471307 (10chasemp) >>! In T140422#2470486, @yuvipanda wrote: > Should get a new key setup for labs root, and probably good to rotate all keys anyway.... [15:54:52] madhuvishy let's do ^ after ops meeting? [16:04:21] YuviPanda: sure [16:37:01] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2471423 (10Dzahn) - Order a yubikey (via OIT?) and set it up ? [16:40:53] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2471431 (10Dzahn) [17:00:28] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2464178 (10mark) Madhu is approved for Ops access. Yes, she should also get a Yubikey. :) [17:03:31] 06Labs, 10Tool-Labs: Normalize kernel on all Debian Jessie nodes in tools - https://phabricator.wikimedia.org/T140611#2471595 (10Aklapper) @Yuvipanda: I guess this is #Tool-Labs ? If not please correct. [17:52:39] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2471900 (10bd808) @dpatrick any status update on this review? [18:19:36] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2471938 (10Dzahn) - added to ops-private - added to root@ mail [18:19:48] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2471939 (10Dzahn) [18:25:24] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2471952 (10chasemp) I asked madhu to change her user key along with the role/access change before https://gerrit.wikimedia.org/r/#/c/299078/. Seems app... [18:51:08] (03PS1) 10Madhuvishy: Add ssh key for user Madhuvishy to root-authorized-keys [labs/private] - 10https://gerrit.wikimedia.org/r/299597 [18:58:44] (03CR) 10Andrew Bogott: [C: 031] Add ssh key for user Madhuvishy to root-authorized-keys [labs/private] - 10https://gerrit.wikimedia.org/r/299597 (owner: 10Madhuvishy) [19:18:32] (03PS2) 10Madhuvishy: Add ssh key for user Madhuvishy to root-authorized-keys [labs/private] - 10https://gerrit.wikimedia.org/r/299597 (https://phabricator.wikimedia.org/T140422) [19:19:56] (03CR) 10Rush: [C: 032] Add ssh key for user Madhuvishy to root-authorized-keys [labs/private] - 10https://gerrit.wikimedia.org/r/299597 (https://phabricator.wikimedia.org/T140422) (owner: 10Madhuvishy) [19:20:05] (03CR) 10Rush: [V: 032] Add ssh key for user Madhuvishy to root-authorized-keys [labs/private] - 10https://gerrit.wikimedia.org/r/299597 (https://phabricator.wikimedia.org/T140422) (owner: 10Madhuvishy) [19:20:38] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2472255 (10chasemp) also: https://gerrit.wikimedia.org/r/#/c/299595 [19:21:07] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2472259 (10chasemp) [19:24:27] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Improvements to cold-migrate script - https://phabricator.wikimedia.org/T139272#2472321 (10Andrew) The password improvements are attached. Deactivate/reactivate are not supported in the current version of the glance client running on labcontrol1001... looks... [19:25:43] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2472333 (10madhuvishy) As for getting a Yubikey - I requested a new one and OIT has place an order. [20:06:53] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2472547 (10dpatrick) Overall, really nice job on entire application. This is a great example of solid Django development. I'm really into striker... [20:07:17] 06Labs, 10Labs-Infrastructure: New tool missing DB replica info - https://phabricator.wikimedia.org/T140677#2472548 (10Magnus) [20:12:22] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2472574 (10chasemp) [20:19:34] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2472619 (10chasemp) [20:34:03] can I get someone to take a look at the instance at deployment-fluorine.deployment-prep.eqiad.wmflabs for me? I can nc port 22 to establish connection, but ssh doesn't let me in. Reboot fixed it once (evidently https://phabricator.wikimedia.org/T140313) tried again with no luck. I think I'm all out of buttons to push on horizon... [20:36:51] chasemp: I don't see the add to ldap/ops on gerrit [20:37:20] ah I thought it was done I'll circle back :) [20:38:27] thcipriani: says active and I can't get on it either [20:38:40] is it possible soemthing is up in hte OS itself? [20:38:44] what's on teh console? [20:38:49] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2472744 (10Dzahn) [20:38:53] chasemp: hmm, can I get you to kick it from nova? I'm not entirely clear that it is *actually* rebooting. [20:39:07] console seems to just show login prompt [20:40:00] yeah kicked from nova [20:40:03] we'll see :) [20:40:20] thanks [20:47:46] chasemp: thanks! I can get in now, digging around for root cause os-side [20:51:06] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Security-Reviews: Security review of Tool Labs console application - https://phabricator.wikimedia.org/T135784#2472840 (10bd808) >>! In T135784#2472547, @dpatrick wrote: > Overall, really nice job on entire application. This is a great example of solid Dja... [20:52:32] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2472852 (10chasemp) [20:59:16] !log tools Disabled puppet on tools-elastic-0[123]. Elasticsearch needs to be upgraded. [20:59:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:00:07] chasemp, andrewbogott, YuviPanda, madhuvishy: if you get notices about puppet being disabled on tools-elastic-0[123] that's me. I should have it all sorted by the end of the day today [21:01:47] 06Labs, 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2472883 (10Dzahn) [21:22:46] kk [21:28:35] 06Labs, 10Labs-Infrastructure: New tool missing DB replica info - https://phabricator.wikimedia.org/T140677#2473027 (10yuvipanda) 05Open>03Resolved a:03yuvipanda I restarted create-dbusers on labstore1001 which seems to have fixed this. [21:30:29] 06Labs, 10Tool-Labs: Normalize kernel on all Debian Jessie nodes in tools - https://phabricator.wikimedia.org/T140611#2473054 (10yuvipanda) Yup! Sorry, I forgot to add project again - something seems to have messed with my muscle memory a bit... [21:33:16] PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:33:27] PROBLEM - Free space - all mounts on tools-worker-1004 is CRITICAL: CRITICAL: tools.tools-worker-1004.diskspace.root.byte_percentfree (<100.00%) [21:34:01] I just turned back on shinken alerts for things [21:35:06] PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:14] PROBLEM - SSH on tools-pastion-01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:42] chasemp can I kill tools-pastion-01? [21:35:49] YuviPanda: yep [21:36:10] goes out w/ closing the bastion flakiness ticket [21:36:18] ok [21:36:20] killing [21:36:37] thanks been meaning to do that [21:37:43] !log tools killed tools-pastion-01, no longer in use [21:37:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:40:15] !log tools bind mount and kill files in /var/lib/docker that were monuted over by proper mount on lvm [21:40:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:40:20] !log tools bind mount and kill files in /var/lib/docker that were monuted over by proper mount on lvm on tools-worker-1004 [21:40:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:40:26] PROBLEM - Host tools-pastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.23.177) [21:40:53] chasemp so tools-precise-dev fails with: [21:40:54] Error: Could not start Service[cgrulesengd]: Execution of '/etc/init.d/cgrulesengd start' returned 2: [21:40:55] Error: /Stage[main]/Cgred/Base::Service_unit[cgrulesengd]/Service[cgrulesengd]/ensure: change from stopped to running failed: Could not start Service[cgrulesengd]: Execution of '/etc/init.d/cgrulesengd start' returned 2: [21:41:14] it's a precise bastion we keep around for peoople who still need it (primarily pytohn users with a virtualenv on precise) [21:41:50] did that always fail? [21:41:59] for a long time at least. [21:42:07] I just am going through shinken failures [21:42:11] I'll take a look [21:42:18] thanks chasemp [21:46:03] YuviPanda: Hey [21:46:21] The uWSGI web service is really stable now :) [21:46:27] Thanks [21:46:39] d3r1ck you realize I haven't touched it at all, right? :) [21:46:55] Really? [21:47:18] YuviPanda: Then maybe its because I am running my codes on venv [21:47:19] yes [21:47:32] Ok, then thank you uwsgi :) [21:47:34] :) indeed, highly recommended that everyone use virtualenvs. [21:47:49] * d3r1ck thanks venv :) [21:48:49] YuviPanda: so what has been keepin you busy lately? :) [21:48:59] * keeping [21:49:55] moving things to kubernetes, mostly [21:50:07] I'll write out a guide for moving uwsgi there, should make restarts faster by an order of magnitude [21:50:15] zhuyifei1999_ is using them just now, but I haven't written docs yet [21:50:41] !log tools force downgrade hhvm on tools-webgrid-lighttpd-1408 to fix puppet issues [21:50:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:53:27] RECOVERY - Free space - all mounts on tools-worker-1004 is OK: OK: All targets OK [22:01:58] YuviPanda: it looks like they changed an option from cap -L to lower -l...ugh and changed realtive rule formats (or revised it in later editions is more true). So I'm considering calling off precise support for bastion cgroupification [22:02:20] I don't think it's worth supporting [22:02:30] chasemp +1 I approve. [22:02:39] can you put a guard around it in puppet? [22:02:44] that's my thought yeah [22:03:21] chasemp want me to file a bug for you? [22:03:25] sure [22:03:33] because I'm about to afk [22:03:35] thanks [22:05:04] RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [22:06:01] 06Labs, 10Tool-Labs: Puppet failures in tools-precise-dev - https://phabricator.wikimedia.org/T140696#2473157 (10yuvipanda) [22:06:03] 06Labs, 10Tool-Labs: Puppet failures in tools-precise-dev - https://phabricator.wikimedia.org/T140696#2473170 (10yuvipanda) a:03chasemp [22:06:04] chasemp ok done [22:06:45] tx [22:07:14] I am done packaging pykube [22:08:14] RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [22:09:52] YuviPanda: Cool :) [22:10:57] now off to sleep :) [22:10:58] good night! [22:11:06] tomorrow, I write a lot more checks for k8s :) [22:35:10] 06Labs, 06Community-Tech-Tool-Labs, 06WMF-Legal: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936#2473347 (10ZhouZ) @bd808 I will publish a new proposed draft of the labs terms of use for community consultation, incorporating the fee... [22:56:40] hey, who is a user in grrrit-wm tool [22:57:17] oh, tool directory https://tools.wmflabs.org/hay/directory/ [22:57:19] checking :) [22:58:21] hmm, cant find it in directory or the list on start page [22:59:57] oh right it's in kubernetes [23:22:55] mutante: https://tools.wmflabs.org/?tool=lolrrit-wm [23:23:16] that's the tool's maintainer list [23:24:28] bd808: oh! i searched for grrrit-wm, thanks for pointing out i have to use lolrrit [23:24:50] it's kind of buried in https://wikitech.wikimedia.org/wiki/Grrrit-wm [23:24:51] i suggest adding paladox to that so he can restart it [23:25:08] i am also not on it.. but "sudo become" works