[00:20:06] <Zppix|mobile>	 I need someone with tools admin rights a dir wont delete
[00:20:14] <Zppix|mobile>	 Ive tried about 5 times
[00:40:44] <shinken-wm>	 PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[01:13:06] <wikibugs>	 06Labs, 10Tool-Labs: Send "are you there?" email to tool labs members every 3 months to revalidate email address - https://phabricator.wikimedia.org/T148792#2733623 (10Platonides) I'm quite sure it didn't, that it was more like once a year (and yet, every time some people failed).
[03:39:13] <wikibugs>	 06Labs, 10Tool-Labs: Send "are you there?" email to tool labs members every 3 months to revalidate email address - https://phabricator.wikimedia.org/T148792#2732932 (10Betacommand) It was every six months and happened via a login/script going to 4 times a year would be kind of obnoxious for most. I have things...
[06:18:50] <shinken-wm>	 PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[06:34:35] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[06:53:51] <shinken-wm>	 RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:14:36] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:17:35] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2733752 (10Marostegui) I had a chat with Jaime yesterday about the past issues with the wildcard-based privileges and it is certainily worrying. Probably...
[07:32:18] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/MikeMel was created, changed by MikeMel link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/MikeMel edit summary: Created page with "{{Tools Access Request |Justification=Goodmorning WikiTech :)    I recently being assigned on a Thesis for my graduate from Technological Educational Institute of Crete (Compu..."
[07:46:14] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Create maintain-views user for labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T148560#2733815 (10Marostegui) So the tricky part is this one:  ```  DEFINER=viewmaster ```  As per MySQL documentation:  ``` If you specify the DEFINER clause, y...
[10:27:41] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Ravikiran.j was created, changed by Ravikiran.j link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Ravikiran.j edit summary: Created page with "{{Tools Access Request |Justification=Learning the git commands. |Completed=false |User Name=Ravikiran.j }}"
[12:20:03] <grrrit-wm>	 (03PS1) 10Jcrespo: labsdb proxy: add fake passwords for sqlproxy [labs/private] - 10https://gerrit.wikimedia.org/r/317144 
[12:20:55] <grrrit-wm>	 (03CR) 10Jcrespo: [C: 032] labsdb proxy: add fake passwords for sqlproxy [labs/private] - 10https://gerrit.wikimedia.org/r/317144 (owner: 10Jcrespo)
[12:21:21] <grrrit-wm>	 (03CR) 10Jcrespo: [V: 032] labsdb proxy: add fake passwords for sqlproxy [labs/private] - 10https://gerrit.wikimedia.org/r/317144 (owner: 10Jcrespo)
[13:42:50] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[13:43:54] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[13:44:12] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1016 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[13:44:20] <shinken-wm>	 PROBLEM - Puppet run on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:44:34] <shinken-wm>	 PROBLEM - Puppet run on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:44:42] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[13:44:46] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1412 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[13:45:20] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:45:22] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[13:45:24] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[13:45:40] <shinken-wm>	 PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[13:45:59] <Alphos>	 why would anyone need tools access to learn git commands ? o_O [12:27:39] <wm-bot> Change on wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/<redacted> was created, changed by <redacted> edit summary: Created page with "{{Tools Access Request |Justification=Learning the git commands. |Completed=false |User Name=<redacted> }}"
[13:46:12] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-gift is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[13:46:45] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1018 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[13:46:51] <shinken-wm>	 PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[13:46:51] <shinken-wm>	 PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[13:47:07] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0]
[13:47:07] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1216 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[13:47:17] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1212 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[13:47:25] <shinken-wm>	 PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:47:25] <shinken-wm>	 PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:47:29] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1017 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[13:47:29] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1220 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[13:47:37] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1217 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[13:47:37] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1418 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:47:38] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[13:47:49] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[13:48:05] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0]
[13:48:47] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[13:49:00] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1011 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[13:49:10] <shinken-wm>	 PROBLEM - Puppet run on tools-grid-master is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[13:49:16] <shinken-wm>	 PROBLEM - Puppet run on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[13:49:28] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[13:50:00] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[13:50:02] <shinken-wm>	 PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[13:50:18] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:50:55] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1006 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[13:51:00] <shinken-wm>	 PROBLEM - Puppet run on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[13:51:02] <shinken-wm>	 PROBLEM - Puppet run on tools-elastic-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[13:51:03] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1020 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[13:51:41] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1209 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[13:51:47] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[13:51:59] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[13:52:19] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[13:52:21] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:52:25] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1420 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[13:52:39] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[13:53:41] <shinken-wm>	 PROBLEM - Puppet run on tools-docker-builder-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[13:54:17] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1207 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[13:54:47] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[13:54:57] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1215 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[13:55:13] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[14:00:25] <Alphos>	 oh god, we're all gonna die, aren't we ? everything's going critical ! :x
[14:02:09] <Zppix|mobile>	 Yes
[14:03:42] * Alphos makes arrangements
[14:04:38] <Zppix|mobile>	 Its ok operations is already blown up...
[14:20:41] <shinken-wm>	 RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:21:51] <shinken-wm>	 RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]
[14:21:53] <shinken-wm>	 RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:22:07] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:22:13] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:22:29] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:22:37] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:22:38] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:22:51] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:23:51] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:24:01] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:24:14] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:24:17] <shinken-wm>	 RECOVERY - Puppet run on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:24:20] <shinken-wm>	 RECOVERY - Puppet run on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:24:26] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:24:34] <shinken-wm>	 RECOVERY - Puppet run on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:24:42] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:24:48] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:25:00] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:25:20] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:25:22] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:25:22] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:25:52] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:26:02] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:26:11] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0]
[14:26:46] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:26:50] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:26:50] <wikibugs>	 06Labs, 06Discovery, 06Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#2734399 (10cmarqu) >>! In T103272#1965254, @akosiaris wrote: >>>! In T103272#1965106, @cmarqu wrote: >> This seems to be down again, e.g. https://tools....
[14:27:04] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:27:20] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:27:24] <shinken-wm>	 RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0]
[14:27:24] <shinken-wm>	 RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:27:26] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:27:30] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:27:38] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:27:40] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:27:50] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:28:06] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:28:40] <shinken-wm>	 RECOVERY - Puppet run on tools-docker-builder-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:28:46] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:29:10] <shinken-wm>	 RECOVERY - Puppet run on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0]
[14:29:16] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1207 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:29:47] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:30:05] <shinken-wm>	 RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:30:17] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:30:19] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:30:59] <shinken-wm>	 RECOVERY - Puppet run on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:31:03] <shinken-wm>	 RECOVERY - Puppet run on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:31:41] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:31:59] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:32:19] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:34:56] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:23:05] <wikibugs>	 06Labs, 06Discovery, 06Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#2734539 (10akosiaris) @madhuvishy Probably an aftermath of the migration ? Is it done (and the only thing required is a VM reboot)? Or not yet ?
[15:29:01] <wikibugs>	 06Labs, 06Discovery, 06Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#2734560 (10madhuvishy) @akosiaris The migration is all done
[15:37:28] <wikibugs>	 06Labs, 06Discovery, 06Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#2734587 (10cmarqu) Works gain, thanks.
[15:46:26] <wikibugs>	 10Striker: Add information about labs-l and labs-annouce to account creation process - https://phabricator.wikimedia.org/T148844#2734605 (10bd808)
[16:03:56] <wikibugs>	 06Labs, 06Discovery, 06Maps: Replacements for a.toolserver.org, b.toolserver.org, c.toolserver.org not available - https://phabricator.wikimedia.org/T103272#2734633 (10akosiaris) I 've restarted maps-tiles3 VM on the maps project for what is worth.
[16:19:00] <wikibugs>	 06Labs: Root emails for projects going to the wrong LDAP groups - https://phabricator.wikimedia.org/T148846#2734659 (10bd808)
[16:23:41] <wikibugs>	 06Labs: Root emails for projects going to the wrong LDAP groups - https://phabricator.wikimedia.org/T148846#2734679 (10Krenair) T138150
[16:35:09] <chippy>	 hello, I'm not seeing /home mounted properly (and subsequent errors saying nfs error in web app) on maps-warper. When I do sudo puppet agent -tv  I see some errors
[16:35:49] <chippy>	 theres a notice Notice: /Stage[main]/Role::Labs::Nfsclient/File[/home]: Not removing directory; use 'force' to override  
[16:35:57] <chippy>	 which may be the reason
[16:39:41] <chippy>	 there is a /home/chippy which is empty. Should I delete it and redo puppet agent?
[16:40:59] <Krenair>	 chippy, did you just start this instance?
[16:41:02] <chippy>	 ahh! https://phabricator.wikimedia.org/T147657
[16:41:27] <chippy>	 Krenair, nope, its been around some time
[16:43:39] <chippy>	 I guess I should comment in that ticket, as it directly mentions Symlinks for /home 
[16:44:41] <chippy>	 lemme reboot it anyhow
[16:48:10] <chippy>	 madhuvishy you seem to be working on this, should I make comment, new issue etc? https://gist.github.com/timwaters/e83e6ab1a75c0bf4248989e0504e3b73  is what puppet agent says
[16:59:31] <madhuvishy>	 chippy: Hi! I couldn't ssh into that instance - logged that none of the updates would be applied there
[16:59:36] <madhuvishy>	 let me look at it now
[16:59:54] <chippy>	 thanks madhuvishy  :)
[17:02:28] <madhuvishy>	 chippy: should be all fixed now
[17:03:16] <madhuvishy>	 !log maps Applied new changes to previously unreachable instance maps-warper.maps (T147657)
[17:03:17] <chippy>	 madhuvishy, thank you! :)
[17:03:17] <stashbot>	 T147657: Move maps share to labstore1003 - https://phabricator.wikimedia.org/T147657
[17:03:20] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL, Master
[17:03:49] <madhuvishy>	 chippy: yw :)
[17:09:55] <chippy>	 yep, it's working again, cheers again :)
[17:10:31] <madhuvishy>	 :D
[17:19:52] <dennyvrandecic>	 if i want to host a bit of static html+javascript in order to provide a small tool for wikidata, is labs the right place?
[17:20:10] <Zppix>	 Tools lab
[17:22:19] <dennyvrandecic>	 is there a how-to on how to do that? looking at this page it looks like I need to run a whole instance just to serve a single html file...
[17:23:25] <dennyvrandecic>	 I mean I love that labs / tool labs is so powerful, but is there a way to do a very simple thing simply? :)
[17:23:47] <Zppix>	 Labs i thought shutdown??
[17:24:08] <dennyvrandecic>	 ok, tool labs
[17:24:29] <Krenair>	 Zppix, what are you talking about?
[17:25:18] <dennyvrandecic>	 maybe tool server
[17:25:39] <dennyvrandecic>	 i mean, maybe zppix refered to tool server
[17:25:42] <Krenair>	 dennyvrandecic, I think you can put files in public_html and run 'webservice start'?
[17:25:54] <Zppix>	 ^^
[17:26:13] <Krenair>	 in tools
[17:27:44] <dennyvrandecic>	 Krenair: thanks, so I don't need to start an instance or something - that sounds good, thanks!
[17:27:57] <wikibugs>	 10Tool-Labs-tools-Xtools: If revisions are revdel'd, articleinfo compares the surrounding edits as if it were one edit - https://phabricator.wikimedia.org/T148857#2734900 (10MusikAnimal)
[17:27:59] <Krenair>	 nah it can be done in tools
[17:28:00] <dennyvrandecic>	 now i need to find my credentials :D
[17:28:24] <dennyvrandecic>	 thx!
[19:14:52] <grrrit-wm>	 (03Draft1) 10Zppix: Test edit to verify gerrit repo can accept non config changes. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/317184 
[19:15:32] <grrrit-wm>	 (03CR) 10Zppix: [C: 032] Test edit to verify gerrit repo can accept non config changes. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/317184 (owner: 10Zppix)
[19:16:36] <grrrit-wm>	 (03Abandoned) 10Zppix: Test edit to verify gerrit repo can accept non config changes. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/317184 (owner: 10Zppix)
[19:38:06] <Betacommand>	 yuvipanda: Sorry if my email came off wrong, I was trying to get an explanation of why 48 hour window is needed. Most IT windows Ive worked with have been less than 12, those that have been longer have had unusual factors that came up
[19:44:05] <annika>	 yuvipanda: afair you helped me (gifti) to set up webservice with k8s. now it's neither working regularly nor with k8s. i have no clue why. are you there to help?
[19:44:57] <doctaxon>	 oh I see, I have tried to start it
[19:45:40] <wikibugs>	 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: labvirt1005 - HP RAID controller issue (battery?) - https://phabricator.wikimedia.org/T148255#2735128 (10Cmjohnson) Swapped disk in slot 0 with new disk   the old disk is being sent back via UPS 1ZA7327E90828184 10
[20:00:58] <yuvipanda>	 Betacommand: thanks! I appreciate the clarification :)
[20:01:10] <yuvipanda>	 Betacommand: I hope madhuvishy's follow up email provided more explanation
[20:01:17] <yuvipanda>	 hi annika 
[20:01:22] <yuvipanda>	 annika: still around?
[20:01:24] <annika>	 hi yuvi
[20:01:44] <madhuvishy>	 Betacommand: I will follow up further with task links etc
[20:02:24] <yuvipanda>	 annika: what issue are you running into?
[20:11:48] <hare>	 https://dumps.wikimedia.org/other/mediacounts/daily/ << are these logs available from Tool Labs?
[20:13:24] <yuvipanda>	 hare: hmm, I can't seem to find it
[20:15:45] <annika>	 yuvipanda: problem seems to be that there are no tcl8.5 libs?
[20:15:50] <annika>	 can you confirm that?
[20:16:05] <yuvipanda>	 annika: which tool is this?
[20:16:10] <annika>	 giftbot
[20:16:24] <yuvipanda>	 looking
[20:17:14] <yuvipanda>	 IOError: [Errno 13] Permission denied: '/var/run/lighttpd/giftbot'
[20:17:18] <yuvipanda>	 seems to be what's happening
[20:17:26] <annika>	 o.o
[20:18:39] <annika>	 yuvipanda: what permissions does it need?
[20:18:40] <yuvipanda>	 annika: I've a fix
[20:18:44] <yuvipanda>	 but gerrit isn't working :(
[20:18:55] <annika>	 ah, ok
[20:19:41] <yuvipanda>	 https://gerrit.wikimedia.org/r/317197 should fix
[20:20:18] <annika>	 ah
[20:22:05] <yuvipanda>	 annika: how about now?
[20:24:37] <annika>	 yuvipanda: nope
[20:24:56] <annika>	 couldn't load file "/data/project/shared/tcl/lib/tcl8.7/tclfcgi0.8/libtclfcgi0.8.so": libtcl8.5.so: cannot open shared object file: No such file or directory
[20:26:26] <yuvipanda>	 interesting
[20:26:52] <yuvipanda>	 that file does exit
[20:26:52] <yuvipanda>	 > -rwxr-xr-x 1 gifti project-tools 13613 Aug 16 00:14 /data/project/shared/tcl/lib/tcl8.7/tclfcgi0.8/libtclfcgi0.8.so
[20:26:54] <yuvipanda>	 and tcl can see it
[20:27:00] <yuvipanda>	 so I don't know what it's confused about
[20:27:31] <annika>	 i think it is libtcl8.5.so
[20:27:40] <annika>	 there is no /usr/bin/tclsh8.5
[20:27:53] <annika>	 so probably it depends on tcl8.5
[20:28:16] <yuvipanda>	 I see
[20:28:18] <yuvipanda>	 well jessie only has https://packages.debian.org/jessie/tcl
[20:28:22] <yuvipanda>	 so that's tcl 8.6
[20:28:27] <yuvipanda>	 plus you aren't using the system provided TCL anyway
[20:28:34] <annika>	 but tclfcgi is
[20:28:36] <yuvipanda>	 you're using your own build, so there isn't much I can do to help there :(
[20:28:50] <annika>	 ok, will build tcl8.5 :D thx
[20:29:13] <yuvipanda>	 ok :)
[20:29:37] <yuvipanda>	 annika: I think we'll be able to support whatever's in jessie, but not so much for custom builds I'm afraid
[20:29:50] <annika>	 yeah, no problem
[20:30:20] <yuvipanda>	 annika: yw! and thanks for reporting the other issue with permissions :)
[20:30:28] <annika>	 yw
[20:55:30] <urandom>	 is this the right place to ask about quotas, or should i file a phabricator?
[20:55:38] <annika>	 yuvipanda: now it complains about libfcgi.so.0
[20:57:02] <yuvipanda>	 urandom: there's a phab
[20:57:18] <urandom>	 yuvipanda: k, should i update https://wikitech.wikimedia.org/wiki/Labs_node_setup?
[20:57:47] <urandom>	 "If you get an error during instance creation, ask in #wikimedia-labs. It's likely that the quota needs to be increased."
[20:58:21] <yuvipanda>	 urandom: yes, and I've also never seen that page before
[20:58:21] <yuvipanda>	 and I think it's specific to deployment-prep
[20:58:52] <yuvipanda>	 it's also wrong in many many other ways
[20:58:57] <yuvipanda>	 such as using role::puppet::self
[21:00:28] <yuvipanda>	 annika: I'm rebuilding to add that, should push itself in <20min
[21:01:27] <annika>	 thx
[21:07:36] <wikibugs>	 06Labs: Request increased quota for services-test labs project - https://phabricator.wikimedia.org/T148869#2735296 (10Eevans)
[21:07:58] <wikibugs>	 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad: labvirt1005 - HP RAID controller issue (battery?) - https://phabricator.wikimedia.org/T148255#2735310 (10Cmjohnson) 05Open>03Resolved a:03Cmjohnson
[21:12:02] <wikibugs>	 10Tool-Labs-tools-Pageviews: Make automatically selecting chart type work without refreshing - https://phabricator.wikimedia.org/T145129#2735318 (10MusikAnimal) Will be fixed in the v2.0 overhaul.
[21:12:40] <wikibugs>	 10Tool-Labs-tools-Pageviews: Pageviews Analysis 2.0 - https://phabricator.wikimedia.org/T147966#2735320 (10MusikAnimal)
[21:12:42] <wikibugs>	 10Tool-Labs-tools-Pageviews: Make automatically selecting chart type work without refreshing - https://phabricator.wikimedia.org/T145129#2735319 (10MusikAnimal)
[21:13:18] <Zppix>	 Hey anyone know if i can git commit from a tools.name from my root user (before typing become)
[21:13:35] <wikibugs>	 10Tool-Labs-tools-Pageviews: Improve Topviews interface - https://phabricator.wikimedia.org/T142802#2735322 (10MusikAnimal) 05Open>03stalled Stalled until the T147966 is released and stable
[21:20:21] <HaeB>	 yuvipanda: do you have a moment for a  PAWS question (kernel died)? zareen here has one
[21:22:33] <zareen>	 Hi yuvi. I keep getting this error "The kernel appears to have died. It will restart automatically.” I was able to dl the tsv file and analyze it on my laptop with Jupyter notebooks but my PAWS kernel keeps dying/restarting when it try it. I’ve restarted the kernel on my own and that didn’t resolve the issue. Here’s my notebook: http://paws-public.wmflabs.org/paws-public/45876923/itwiki_section_frequency
[21:22:34] <zareen>	 .ipynb
[21:22:49] <zareen>	 http://paws-public.wmflabs.org/paws-public/45876923/itwiki_section_frequency.ipynb
[21:23:38] <zareen>	 after the error message (when the kernel should have automatically restarted), any cell i run just runs indefinitely 
[21:26:09] <wikibugs>	 06Labs, 10Tool-Labs: Make webservice command read default cli arguments from ~/.webservicerc - https://phabricator.wikimedia.org/T148872#2735362 (10bd808)
[21:26:50] <annika>	 yuvipanda: shall i still be waiting for the file to appear?
[21:32:54] <Zppix>	 Why cant i commit to gerrit repo that i have owner perms on from a tool.account
[21:45:10] <Zppix>	 ??
[21:45:56] <paladox>	 Zppix, link please?
[21:46:11] <paladox>	 Plus what error are you getting?
[21:46:13] <Zppix>	 Link to what?
[21:46:21] <Zppix>	 Its saying auth failed
[21:46:48] <paladox>	 Link to the gerrit repo
[21:46:49] <paladox>	 and oh
[21:48:56] <Zppix>	 I thinks it cause im a dumbfuck
[21:49:01] <Zppix>	 And sued anom http
[21:49:12] * Zppix facepalm
[21:50:13] <paladox>	 oh
[21:55:56] <shinken-wm>	 RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms
[22:01:02] <shinken-wm>	 PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170)
[22:03:10] <Zppix>	 paladox: why is git review asking for username and pass??
[22:03:27] <paladox>	 Zppix you may be using http
[22:03:31] <paladox>	 check your remote in
[22:03:35] <paladox>	 .git/config
[22:03:45] <Zppix>	 I changed it to ssh
[22:03:52] <paladox>	 Oh check it again
[22:04:14] <Zppix>	 I did
[22:04:17] <Zppix>	 Twice
[22:05:10] <paladox>	 Oh, not sure then
[22:05:41] <paladox>	 Zppix try git remote --verbose
[22:05:43] <Zppix>	 Whats gitreview supposed to be located with repo files or .git
[22:06:05] <paladox>	 Zppix .git/config is located under the repo you just cloned
[22:06:14] <Zppix>	 No errors
[22:06:15] <paladox>	 for example MediaWiki core is core/.git/config
[22:06:15] <Zppix>	 No
[22:06:21] <Zppix>	 Gitreview
[22:06:22] <Zppix>	 File
[22:06:24] <paladox>	 git remote --vv is to show you your remotes
[22:06:40] <paladox>	 Im unsure what gitreview about that, i doint use gitreview
[22:06:46] <paladox>	 Maybe -operations will know
[22:06:49] <Zppix>	 Both ssh
[22:06:49] <paladox>	 some ops users use it
[22:07:12] <Zppix>	 paladox: to submit to gerrit i think it requires gitreview xd
[22:07:39] <paladox>	 Zppix no, you can use plain git or git-review
[22:07:48] <paladox>	 many users prefer plain git over git-review
[22:08:01] <paladox>	 I use plain git and can submit to gerrit and have done so for the last 3+ years
[22:09:03] <Zppix>	 How do u use git instead review
[22:09:24] <paladox>	 Zppix, you just do git clone <url>
[22:09:29] <paladox>	 and then git add -A --all
[22:09:32] <paladox>	 git commit
[22:09:46] <paladox>	 then git push origin HEAD:refs/for/<branch>
[22:09:58] <paladox>	 ^^ that push is for getting changes to be landed on gerrit for review
[22:10:09] <paladox>	 git push origin HEAD:refs/heads/<branch>
[22:10:27] <paladox>	 ^^ that's for landing the change straight into the repo without getting the changes reviewed
[22:16:20] <d3r1ck>	 paladox: Hello :)
[22:16:25] <paladox>	 Hi
[22:16:56] <d3r1ck>	 I am glad the changes were deployed :)
[22:17:06] <d3r1ck>	 No more unusual "Or"
[22:17:19] <d3r1ck>	 +2 for doing that :)
[22:17:36] <paladox>	 Oh
[22:17:41] <wikibugs>	 06Labs: Request increased quota for services-test labs project - https://phabricator.wikimedia.org/T148869#2735445 (10Eevans)
[22:17:52] <paladox>	 Which one?
[22:18:25] <d3r1ck>	 I filled an issue yesterday about commons showing an unusual "Or" when on the Upload File page
[22:18:52] <d3r1ck>	 paladox: Remember now?
[22:20:01] <paladox>	 Oh
[22:20:03] <paladox>	 yes
[22:20:04] <paladox>	 Yep :)
[22:20:17] <d3r1ck>	 Ok cool
[22:20:25] <d3r1ck>	 Just checked it and its gone :)
[22:28:22] <shinken-wm>	 RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.85 ms
[22:34:24] <shinken-wm>	 PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22)
[22:36:28] <yuvipanda>	 HaeB: zareen hey! still around?
[22:36:44] <yuvipanda>	 annika: hey! if you do webservice restart, it should appear now
[22:37:02] <paladox>	 :)
[22:37:06] <wikibugs>	 06Labs, 10Tool-Labs: Make webservice command read default cli arguments from ~/.webservicerc - https://phabricator.wikimedia.org/T148872#2735362 (10yuvipanda) It already reads them from service.manifest...
[22:37:31] <HaeB>	 yuvipanda: i think zareen is afk currently, but will surely be able to read scrollback
[22:41:08] <shinken-wm>	 RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.91 ms
[22:43:51] <annika>	 yuvipanda: yay, woohoo, works
[22:44:15] <yuvipanda>	 HaeB: hmm I can't seem to reproduce it
[22:44:21] <yuvipanda>	 zareen: can you go to 'control panel' in top right, 'stop server', and 'start server' again?
[22:45:22] <Zppix>	 yuvipanda: is it possible to git review to gerrit via shell on my tool.account
[22:45:35] <yuvipanda>	 Zppix: hmm, I don't know
[22:45:43] <yuvipanda>	 Zppix: I think you'll have to use https to push
[22:45:47] <yuvipanda>	 not sure how
[22:45:49] <yuvipanda>	 Zppix: I think legoktm has used it in the past
[22:45:52] <Zppix>	 Tried
[22:45:56] <Zppix>	 Https doesnt work
[22:46:12] <Zppix>	 It throws fatals and auth errors
[22:46:42] <yuvipanda>	 then I've no idea
[22:46:43] <yuvipanda>	 sorry
[22:46:58] <paladox>	 Zppix did you add an ssh key
[22:47:27] <Zppix>	 Theres one in .ssh on my root acct and tool
[22:47:30] <paladox>	 You will need to generate the key on the instance your on, the ssh key wont be the one you used to ssh in since that wont work
[22:47:34] <paladox>	 oh root.
[22:47:38] <paladox>	 not sure then
[22:48:07] <shinken-wm>	 PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218)
[22:55:15] <yuvipanda>	 Zppix: paladox please do not put ssh private keys on tools
[22:55:19] <yuvipanda>	 we go through them ocassionally and delete them
[22:55:29] <yuvipanda>	 it can be used to impersonate you and we don't want to have that security risk on tools or on labs
[22:55:34] <paladox>	 yuvipanda i doint store ssh keys on las
[22:55:35] <paladox>	 labs
[22:55:37] <paladox>	 or tools
[22:55:42] <Zppix>	 Ya i deleted them
[22:55:48] <yuvipanda>	 ok :)
[22:57:06] <paladox>	 :)
[22:57:14] <wikibugs>	 06Labs, 10Tool-Labs: Make webservice command read default cli arguments from ~/.webservicerc - https://phabricator.wikimedia.org/T148872#2735535 (10bd808) Doesn't that file get wiped out when you `webservice stop`? I thought there was some need to remember to use `--backend=kubernetes` each time you started th...
[23:00:04] <bd808>	 Zppix: I have used https with git-review and with raw git from labs host. You use the username/password from https://gerrit.wikimedia.org/r/#/settings/http-password. Labs hosts aren't trustworthy for secrets, so each time I use a password I then hit "generate password" to change it.
[23:00:48] <bd808>	 I think that you need a sort of new version of git-review for it to work properly with an https origin. I tend to install it in a virtualenv.
[23:10:51] <d3r1ck>	 Ohhhh no, NetSplit is messing up :)
[23:44:50] <shinken-wm>	 PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[23:53:51] <wikibugs>	 06Labs, 07LDAP: Clean up ldap host entries and references - https://phabricator.wikimedia.org/T148781#2735613 (10Krenair)