[00:18:43] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1029 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[00:21:49] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1028 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[01:40:06] <wikibugs_>	 10Labs-project-other: Successful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#3100692 (10Samwilson) If this is going to be closed, then I guess the subtasks should also be closed as invalid? And the discourse install...
[01:59:41] <TParis>	 anomie or bd808 ping, please help
[02:03:58] <wikibugs_>	 10Labs-project-other: Successful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#3100697 (10Tgr) 05Invalid>03Open Closing as invalid does not seem to make much sense. Either the pilot is done (and I'd rather leave i...
[06:40:37] <wikibugs_>	 (03PS7) 10Zhuyifei1999: Add rewritten crontab in Python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174)
[06:42:24] <wikibugs_>	 (03CR) 10Zhuyifei1999: [C: 04-1] "Rebased; WIP" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999)
[06:50:22] <shinken-wm_>	 PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[07:01:36] <wikibugs_>	 (03PS8) 10Zhuyifei1999: Add rewritten crontab in Python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174)
[07:13:45] <wikibugs_>	 (03CR) 10Zhuyifei1999: [C: 04-1] "I guess I'll remove the crlino logic. It's difficult to add the tests with some states variables lying around without OOP." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999)
[07:25:22] <shinken-wm_>	 RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[08:43:36] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=enwiktionary --site-group=wiktionary // (1 site added)
[08:43:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[08:43:51] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=dewiktionary --site-group=wiktionary // (0 sites added)
[08:43:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[08:44:51] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=enwiktionary --force-protocol=https
[08:44:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[08:49:12] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript sql.php --wiki=enwiktionary "TRUNCATE sites; TRUNCATE site_identifiers;"
[08:49:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[08:49:36] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=enwiktionary --force-protocol=https --load-from=https://deployment.wikimedia.beta.wmflabs.org/w/api.php
[08:49:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[08:50:33] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki=enwiktionary --site-group=wiktionary // (3 sites added)
[08:50:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[08:56:05] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=enwiktionary // (ParameterTypeException, T160503)
[08:56:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[08:56:12] <stashbot>	 T160503: Wikimedia\Assert\ParameterTypeException from line 89 of /srv/mediawiki-staging/php-master/vendor/wikimedia/assert/src/Assert.php: Bad value for parameter $namespace: must be a integer - https://phabricator.wikimedia.org/T160503
[09:08:13] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=enwiktionary
[09:08:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[09:10:00] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=dewiktionary
[09:10:03] <addshore>	 !log deployment-prep addshore@deployment-tin mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki=hewiktionary
[09:10:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[09:10:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[09:31:48] <wikibugs_>	 (03CR) 10Zhuyifei1999: Add rewritten crontab in Python (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999)
[10:01:24] <wikibugs_>	 (03PS9) 10Zhuyifei1999: Add rewritten crontab in Python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174)
[10:11:58] <wikibugs_>	 (03CR) 10Zhuyifei1999: Add rewritten crontab in Python (032 comments) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/336998 (https://phabricator.wikimedia.org/T156174) (owner: 10Zhuyifei1999)
[10:15:05] <wikibugs_>	 06Labs, 10Tool-Labs: Support @weekly et al in crontab - https://phabricator.wikimedia.org/T86446#3101315 (10zhuyifei1999) For reference, the regex may not be good enough and this could be still an issue. See https://gerrit.wikimedia.org/r/#/c/336998/9/misctools/oge-crontab
[10:28:54] <wikibugs_>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3101338 (10Petrb) Hello,  wm-bot is depending on huggle-pg instance which was not yet migrated, it's not so easy, it's a postgres database, and it's pretty huge. Hopefully I will be able to resolve this this week, I am...
[10:33:07] <wikibugs_>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3101346 (10Petrb) I mean, if huggle-pg is down, wm-bot itself will be still operational, but we lose access to SQL based IRC logs. We probably don't want that, the database is kinda useful and valuable, problem is that...
[10:37:04] <wikibugs_>	 06Labs, 10Huggle: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted - https://phabricator.wikimedia.org/T157710#3101385 (10Petrb) @Andrew I tried to reach you on IRC with no luck, can you get to huggle instance? I ran dist-upgrade there and the SSH session timed out and now I can't log back i...
[11:11:48] <wikibugs_>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3101423 (10Paladox) @Petrb what about migrating a postgress db to a mysql one? Will that work? mysql is advanced enough to support online migration as you won't need to take the db down.
[12:33:24] <wikibugs_>	 06Labs, 10wikitech.wikimedia.org, 07WMF-maintenance-script-run: migrateUserGroups.php after T158039 - https://phabricator.wikimedia.org/T158416#3101565 (10MarcoAurelio)
[12:34:24] <wikibugs_>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3101581 (10Petrb) I am not aware of any such feature and highly doubt it.  We got rid of mysql which was originally used, because of lack of many features that enterprise-grade rdbms like postgre or oracle provide, mov...
[12:35:24] <wikibugs_>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3101597 (10Petrb) also I don't see how you could migrate from postgres to mysql online, that also isn't possible, so this definitely would not work.
[12:37:47] <wikibugs_>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3101649 (10Petrb) probably most easy way to do this would be to switch wm-bot to newly created DB on wm-bot-pg and then start migration of existing data, I will have a look into this later, but I am too busy now. I am...
[12:52:31] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Javad was created, changed by Javad link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Javad edit summary: Created page with "{{Tools Access Request |Justification=runnig python bot (JYBot) on fa.wikipedia |Completed=false |User Name=Javad }}"
[13:51:22] <shinken-wm_>	 PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[14:07:28] <anomie>	 Hmm. Tool Labs is being slow this morning.
[14:12:43] <chasemp>	 anomie: slow in what way? working on teh bastion?
[14:13:08] <anomie>	 chasemp: On the bastion, yes.
[14:14:29] <anomie>	 chasemp: It may be filesystem (NFS?) related, CPU-bound tasks don't seem to be slow.
[14:16:52] <chasemp>	 anomie: likely, I'll take a look
[14:18:10] <chasemp>	 wut
[14:18:10] <chasemp>	 tools.m+ 13846 13683  9 11:15 pts/28   00:17:45 java -Xmx256M -Dcom.android.sdkmanager.toolsdir=/data/project/mjbmr/.sys/android-sdk/tools -classpath /data/project/mjbmr/.sys/android-sdk/tools/lib/sdkmanager.jar:/data/project/mjbmr/.sys/android-sdk/tools/lib/swtmenubar.jar:/data/project/mjbmr/.sys/android-sdk/tools/lib/x86_64/swt.jar com.android.sdkmanager.Main update sdk -u -a
[14:20:07] <chasemp>	 anomie: any better?
[14:20:20] <anomie>	 chasemp: Much, thanks
[14:31:23] <shinken-wm_>	 RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:32:01] <wikibugs_>	 06Labs, 10Huggle: Labs instance huggle.huggle.wmflabs needs to be replaced or deleted - https://phabricator.wikimedia.org/T157710#3102467 (10Andrew) @Petrb, I completed the in-place upgrade on Huggle and it looks ok to me...  andrew@huggle:~$ uname -a Linux huggle 3.13.0-112-generic #159-Ubuntu SMP Fri Mar 3 1...
[15:50:03] <chasemp>	 !log tools (late) kill what appears to be an android emulator? unsure but it's eating all IO
[15:50:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:04:03] <zhuyifei1999_>	 chasemp: would it make sense to change the past tense of "Precise tools will fail starting" into something like "are failing"?
[16:04:12] <chasemp>	 zhuyifei1999_: sure
[16:04:28] <chasemp>	 I do enjoy tense accuracy :)
[16:04:45] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3102678 (10yuvipanda)
[16:04:49] <wikibugs_>	 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#3102676 (10yuvipanda) 05Open>03Resolved THE PLEASURE IS ALL MINE!
[16:08:47] <chasemp>	 !log stop puppet on k8s master and drain nodes
[16:08:48] <stashbot>	 Unknown project "stop"
[16:08:50] <chasemp>	 !log tools stop puppet on k8s master and drain nodes
[16:08:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:11:12] <chasemp>	 !log tools k8s master 'for h in `kubectl get nodes | grep worker | grep -v NotReady | grep -v Disabled | awk '{print $1}'`; do echo $h && kubectl drain --delete-local-data --force $h && sleep 10 ; done'
[16:11:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:11:55] <Zppix>	 wait are you killing k8s
[16:12:47] <Reedy>	 upgrades?
[16:12:51] <Reedy>	 (guess)
[16:13:16] <yuvipanda>	 Zppix: yes, we emailed labs-announce about it last week
[16:13:29] <chasemp>	 https://lists.wikimedia.org/pipermail/labs-announce/2017-March/000220.html
[16:13:36] <Zppix>	 i am only subscribed to wikitech-l
[16:13:53] <chasemp>	 everyone who does anything in labs should be subscribed to labs-announce just fyi
[16:22:16] <chasemp>	 !log tools service kube-apiserver stop; service kube-scheduler stop; service kube-controller-manager stop
[16:22:46] <bd808>	 chasemp: heh. hard to !log when stashbot's pod has gone missing
[16:22:53] <Zppix>	 LOL
[16:23:02] <chasemp>	 heh
[16:23:22] <Zppix>	 what do you do when you kill k8s you try to use a k8s using bot :P
[16:50:13] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1021 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[16:54:45] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1010 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[16:55:07] <shinken-wm_>	 PROBLEM - Puppet run on tools-worker-1026 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0]
[17:00:14] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:04:49] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:05:07] <shinken-wm_>	 RECOVERY - Puppet run on tools-worker-1026 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:10:58] <wikibugs_>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Deprecate precise instances in Labs by 2017-03-31 - https://phabricator.wikimedia.org/T143349#3103001 (10Andrew)
[17:20:45] <chasemp>	 !log tools test of logging
[17:20:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:25:20] <wikibugs_>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3103068 (10tom29739) The easiest and safest solution would probably be using something like [[ https://bucardo.org/wiki/Bucardo | Bucardo ]] ([[ https://www.compose.com/articles/using-bucardo-5-3-to-migrate-a-live-post...
[17:29:54] <chasemp>	 !log tools docker stop && rm -fR /var/lib/docker/* on worker-1001
[17:29:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:36:19] <wikibugs_>	 06Labs, 07LDAP: Purge stale data from LDAP - https://phabricator.wikimedia.org/T138150#3103175 (10Andrew)
[17:36:21] <wikibugs_>	 06Labs, 07LDAP, 13Patch-For-Review: Clean up ldap host entries and references - https://phabricator.wikimedia.org/T148781#3103173 (10Andrew) 05Open>03Resolved I removed all of the ldap host entries.
[17:36:41] <halfak>	 !log ores deployed ores-wmflabs-deploy:080418d to staging
[17:36:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[17:42:04] <madhuvishy>	 !log tools Restarted stashbot
[17:42:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:57:13] <wikibugs_>	 06Labs, 10WM-Bot: Move wm-bot instance to Trusty - https://phabricator.wikimedia.org/T157838#3103314 (10Paladox) What about MariaDB?
[18:16:42] <wikibugs_>	 06Labs, 10video2commons: Can we delete these old files on video project? - https://phabricator.wikimedia.org/T158786#3103432 (10madhuvishy) @Matanya Status update on this cleanup?
[19:03:52] <halfak>	 !log ores deployed ores-wmflabs-deploy:080418d
[19:03:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[19:13:40] <SMalyshev>	 bd808: do you know anything about new vagrant image having problems with openssl?
[19:14:17] <SMalyshev>	 bd808: I've installed it and it produced tons of errors until I manually upgraded openssl
[19:21:15] <andrewbogott>	 !log tools added new exec nodes:  tools-exec-1421 and tools-exec-1422
[19:21:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:24:05] <halfak>	 !log ores staged ores-wmflabs-deploy:0990b1d
[19:24:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[19:26:14] <halfak>	 !log ores deployed ores-wmflabs-deploy:0990b1d
[19:26:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL
[19:31:00] <danilo>	 tomthirteen: ask here your doubts, this is the right channel
[19:35:22] <danilo>	 tomthirteen: see this section, maybe it help https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Using_Tool_Labs_and_managing_your_files
[19:35:36] <danilo>	 and ask here, not in -en-help
[19:39:18] <chippy>	 Hello, I want to switch the proxy for maps-warper to point to maps-warper2   - I think I can do this in horizon by deleting the existing and creating a new record. Is this the right way, anything I need to do before / after? 
[19:39:35] <chippy>	 ^ i mean for warper.wmflabs.org 
[19:40:24] <chasemp>	 chippy: you should be able to change teh record regardless of deleting the old instance
[19:40:34] <chasemp>	 it's all manageable in horizon.wikimedia.org I believe
[19:40:37] <chasemp>	 yes
[19:40:47] <chasemp>	 andrewbogott: ^ is that right?
[19:41:07] <chippy>	 chasemp, thanks - yeah, I've got one proxy entry for warper.wmflabs  for maps-warper. and the button just says delete
[19:41:24] <chippy>	 so before I remove the actual instance, I can remove the proxy dns entry
[19:41:45] <chippy>	 and then create a new proxy entry. 
[19:41:50] <chasemp>	 sure yes the proxy entry is managed separately is what I meant, you sould be abel to delete and recreate pointing at new instance 
[19:41:55] <chasemp>	 outside of removing the old instance
[19:41:57] <chippy>	 great stuff, thanks!
[19:42:08] <chippy>	 i was reading on wikitech and it seemed more complex 
[19:43:15] <chippy>	 yeah, I won't be deleting the old instance just yet... have to be doubly sure and do a good search to see if theres stuff on in I need to keep (logs etc) but I think tomorrow I will be able to confirm that the new instance is working.
[19:50:55] <andrewbogott>	 chasemp, chippy, yes, proxies can be managed in Horizon, I think it's straightforward.
[19:51:04] <chippy>	 thanks andrewbogott, chasemp 
[19:52:01] <chippy>	 oh, i came across an issue with the ubuntu 14 wikimedia repository having an old version of some library, is that worth me making a phabricator ticket for?
[19:55:06] <andrewbogott>	 chippy: I'm not sure I understand...
[19:55:13] <andrewbogott>	 but, sure, you can make a ticket.  Is it currently a problem for you?
[19:55:25] <chippy>	 not currently, solved it after 2 hours!
[19:56:03] <chasemp>	 we don't generally upgrade packages outside of the main repos, but pulling from backport for your own purposes on your own project is reasonable
[19:56:31] <chippy>	 the issue was that the ubuntu 14.04 image has a wikimedia repository enabled 
[19:57:34] <chippy>	 so for icu-devtools apt.wikimedia was offering  4.8.1.1-14+trusty2 but the main repo was 52.1-3
[19:58:06] <chippy>	 so when I did apt install icu-devtools it was getting the lower version number 
[19:58:19] <bd808>	 SMalyshev: was it openssl inside the VM or openssl on your host computer?
[19:58:38] <bd808>	 either way I've not seen it before but I imagine the fixes would be different
[20:13:10] <SMalyshev>	 bd808: inside vagrant
[20:13:42] <SMalyshev>	 bd808: looks like some dependency is messed up, clean install just fails but after I manually upgrade openssl everything is ok
[20:25:46] <bd808>	 SMalyshev: weird. I wonder if you have a role that is pulling in something that needs openssl from the WMF apt repo but somehow its not getting installed automatically?
[20:37:43] <wikibugs_>	 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3103909 (10chasemp) https://lists.wikimedia.org/pipermail/labs-announce/2017-March/000221.html  ```Complete.  This maintenance lasted just shy of an hour.  All Kuber...
[20:45:14] <matanya>	 madhuvishy: are you around ?
[20:45:53] <halfak>	 O
[20:45:58] <madhuvishy>	 matanya: hi! yes
[20:46:10] <halfak>	 I'm getting a 502 for PAWS.  Anyone else?  
[20:46:30] <matanya>	 madhuvishy: the files i have are only around 83GB, how did you get to 2TB ?
[20:46:50] <matanya>	 can you point me to the offensive ones ?
[20:47:22] <bd808>	 halfak: it seems to be working for me
[20:47:36] <halfak>	 I only see the issue when I try to start my server
[20:47:43] <halfak>	 I can load up https://paws.wmflabs.org/paws/hub/home
[20:48:35] <bd808>	 hmmm.. I stopped my server and started again and hit a redirect error in Firefox
[20:49:23] <bd808>	 I think for me that's just the old special chars in username bug though
[20:49:50] <madhuvishy>	 https://www.irccloud.com/pastebin/c5VjjEsG/
[20:49:53] <bd808>	 going back to paws.wmflabs.org seems to fix it for me
[20:49:57] <madhuvishy>	 matanya: ^^
[20:50:13] <madhuvishy>	 1.9T	video/project/wikimania2014
[20:50:37] <matanya>	 ah, that can be safely deleted
[20:50:49] <matanya>	 also the old craft dir can
[20:51:09] <matanya>	 regarding ready i need to check
[20:51:48] <madhuvishy>	 matanya: those are much smaller - removing wikimania2014 itself would help a lot
[20:52:04] <matanya>	 madhuvishy: it is owned by root
[20:52:13] <bd808>	 halfak: so I'm not saying your are crazy, but I can't reproduce
[20:52:27] <matanya>	 yes, ready can go as well
[20:52:37] <halfak>	 bd808, thanks for checking.  Just to confirm, you can shut down your server and restart it?
[20:53:20] <matanya>	 madhuvishy: deleting now
[20:53:28] <bd808>	 halfak: yes. I did get a weird infinite redirect when it started, but then going back to the main url fixed that and I could get a shell session etc
[20:53:29] <madhuvishy>	 okay
[20:54:50] <wikibugs_>	 06Labs, 10video2commons: Can we delete these old files on video project? - https://phabricator.wikimedia.org/T158786#3103958 (10Matanya) deleting now.
[20:55:42] <madhuvishy>	 matanya: <3 thank you
[20:55:53] <matanya>	 madhuvishy: sure, all done
[20:56:56] <wikibugs_>	 06Labs, 10video2commons: Can we delete these old files on video project? - https://phabricator.wikimedia.org/T158786#3103967 (10Matanya) Before:  ``` df -h Filesystem                                                Size  Used Avail Use% Mounted on udev                                                       10M...
[20:57:27] <wikibugs_>	 06Labs, 10video2commons: Can we delete these old files on video project? - https://phabricator.wikimedia.org/T158786#3103968 (10Matanya) 05Open>03Resolved
[20:57:41] <madhuvishy>	 awesome
[20:57:50] <matanya>	 anything else madhuvishy ?
[20:57:59] <madhuvishy>	 matanya: nope all good :) thank you
[20:58:06] <matanya>	 I come to serve
[20:58:13] <halfak>	 bd808, thanks.  Not sure what I should try next.  Maybe file a phab task?
[20:58:38] <halfak>	 Oh wait!  I'm not stuck on infinite redirects
[20:58:41] <halfak>	 *now
[20:59:09] <bd808>	 what happens if you go back to https://paws.wmflabs.org/ ?
[20:59:27] <halfak>	 Only a "Start My Server" button
[20:59:35] <bd808>	 hmmm...
[20:59:47] <halfak>	 Tried logging out and re-oauthing
[20:59:49] <halfak>	 No difference
[21:00:14] <bd808>	 Sounds like time for a phab task and then to see if madhuvishy or yuvipanda can track down some logging
[21:00:23] * bd808 doesn't know where to look
[21:00:32] <madhuvishy>	 halfak: i'll look now
[21:00:52] <halfak>	 thanks madhuvishy.  Should I still make a task or wait for a minute?
[21:01:33] <madhuvishy>	 halfak: if i can't find anything, let's make a task - hold on for a moment :)
[21:01:50] <halfak>	 kk
[21:12:31] <madhuvishy>	 halfak: try now?
[21:12:34] <madhuvishy>	 i'm watching logs
[21:19:02] * halfak clicks "STart My Server"
[21:19:05] <halfak>	 It worked!
[21:19:11] <madhuvishy>	 yay
[21:19:13] <halfak>	 madhuvishy, what did you do?
[21:19:25] <madhuvishy>	 i deleted your user's kubernetes pod
[21:19:42] <madhuvishy>	 wonder if it got borked during the migration this morning somehow
[21:19:52] <madhuvishy>	 it respawned now and things seem good
[21:20:09] <halfak>	 Cool with me.  Thanks for your help :) 
[21:20:19] <madhuvishy>	 halfak: no problem :
[21:20:21] <madhuvishy>	 :)
[21:20:23] * halfak wonders if he can get an "unbork the k8s" button
[21:20:52] <halfak>	 I'll remember to file that one on April 1st
[21:20:56] <halfak>	 ;) 
[21:20:56] <madhuvishy>	 halfak: ha ha it's a bit hard to know what exactly to unbork. in this case only your user was having the redirect on log in trouble
[21:21:00] <madhuvishy>	 ;)
[21:21:14] <madhuvishy>	 everyone else was fine. sometimes we've to unbork it all
[21:28:21] <halfak>	 Oooh.  I want that button. 
[21:28:42] <halfak>	 "unbork everything for everyone by shutting everything down now" 
[21:28:45] <halfak>	 great idea
[21:30:12] <madhuvishy>	 halfak: he he for real that's what we do sometimes ;)
[21:44:17] <wikibugs_>	 06Labs, 10wikitech.wikimedia.org: Get rid of SemanticMediaWiki/SRF/SF from wikitech.wikimedia.org - https://phabricator.wikimedia.org/T53642#3104109 (10bd808)
[21:44:27] <wikibugs_>	 10Quarry: Quarry runs thousands times slower in last months - https://phabricator.wikimedia.org/T160188#3091603 (10Halfak) I'm also noticing that most queries stay queued now.  Running queries directly against LabsDBs results in far better performance so it might be something in Quarry.  Celery maybe.
[21:45:30] <SMalyshev>	 bd808: looks like openssl is installed but wrong version: /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0: version `OPENSSL_1.0.1s' not found (required by openssl)
[21:47:06] <bd808>	 hmmm... that sounds like a Debian base image sort of problem. Would you mind filing a bug about it? I'll see if I can figure out how to fix that.
[21:47:44] <Platonides>	 openssl in jessie is at 1.0.1t-1+deb8u6
[21:48:20] <Platonides>	 so it wouldn't be a problem
[21:52:05] <wikibugs_>	 10Striker: Allow easy replication of existing github/bitbucket repos - https://phabricator.wikimedia.org/T143971#3104123 (10mmodell)
[21:52:10] <wikibugs_>	 10Striker, 10Phabricator, 10Security-Reviews, 13Patch-For-Review: Unable to mirror repository from git.legoktm.com into diffusion - https://phabricator.wikimedia.org/T143969#3104124 (10mmodell)
[21:57:13] <shinken-wm_>	 PROBLEM - Puppet run on tools-exec-1424 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0]
[22:02:18] <shinken-wm_>	 RECOVERY - Puppet run on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:51:53] <tomthirteen>	 hello all
[22:52:00] <tomthirteen>	 Could someone please help me?
[22:52:22] <tomthirteen>	 I have set up a labs account
[22:52:28] <tomthirteen>	 under tomthirteen
[22:52:42] <tomthirteen>	 but I cannot save anything to this directory
[22:52:57] <tomthirteen>	 I keep getting  ERROR 1045 (28000): Access denied for user 'u10735'@'%' (using password: YES) MariaDB [simplewiki_p]>
[22:53:24] <tomthirteen>	 I obviously need permission, but I can't figure out how to?
[22:57:41] <bd808>	 tomthirteen_: the database replicas are read-only. You will not be able to change the contents of the simplewiki database directly.
[22:58:49] <tomthirteen_>	 Yes, but can't I use the directory given to me to run sql queries? 
[23:00:02] <bd808>	 you are showing both an error and a successful login prompt, so I guess I'm confused about what is failing
[23:00:34] <bd808>	 the 'directory' is really a unix user account but I understand that part
[23:00:37] <wikibugs_>	 06Labs, 10Tool-Labs: Access Denied - https://phabricator.wikimedia.org/T160585#3104382 (10Tomthirteen)
[23:00:55] <jynus>	 ah
[23:01:05] <jynus>	 select ...into outfile
[23:01:16] <bd808>	 and your user account should have a $HOME/replica.my.cnf 
[23:01:24] <bd808>	 ah. yeah that won't work
[23:01:41] <tomthirteen_>	 Ok, I understand that the databases are read only, but my understanding is that I can understand how to run and save sql queries
[23:01:54] <jynus>	 tomthirteen_, you are trying to write a file to the database
[23:01:57] <jynus>	 that will not work
[23:01:58] <bd808>	 jynus: do you have a good link on things other than `into outfile`
[23:02:11] <jynus>	 https://mariadb.com/kb/en/mariadb/select-into-outfile/
[23:02:52] <jynus>	 just run mysql -e "select distinct pr_page from page_restrictions limit 100" > data/project/tomthirteen/test.txt (with the appropiate options) instead
[23:03:31] <jynus>	 you cannot use select into outfile for a remote database
[23:04:39] <jynus>	 https://dev.mysql.com/doc/refman/5.7/en/select-into.html may be better and more explained
[23:06:47] <tomthirteen_>	 I tried mysql -e "select distinct pr_page from page_restrictions limit 100" > data/project/tomthirteen/test.txt 
[23:06:52] <tomthirteen_>	 this doesn't work
[23:06:55] <jynus>	 of course
[23:07:18] <jynus>	 you need to add extra options to connect
[23:07:26] <jynus>	 the same way you connected before
[23:08:16] <bd808>	 tomthirteen_: try this -- sql simplewiki_p -e "select distinct pr_page from page_restrictions limit 100" > test.txt
[23:08:24] <jynus>	 the idea is to use -e to specify a query and redirection pipe (standard output) ">" to a file
[23:09:01] <tomthirteen_>	 WORKED!!!
[23:09:06] <jynus>	 bd808, thanks, I am not familiar with the env labs users have
[23:09:33] <bd808>	 jynus: yeah, no worries! You helped lots by seeing the error on the server logs :)
[23:09:47] <jynus>	 I didn't
[23:09:47] <tomthirteen_>	 Thanks!
[23:10:14] <jynus>	 I was about to go to sleep when I saw permission denied and kept around
[23:10:14] <bd808>	 just omniscient jynus ?
[23:11:17] <jynus>	 who can avoid the possibitliy of helping someone with mysql problems?
[23:11:31] <jynus>	 today I did my work, I helped on stackoverflow, and on reddit and on IRC
[23:11:33] <tomthirteen_>	 ok so I need to do sql database to "use" the database
[23:11:39] <bd808>	 :) this is the place to find that kind of work
[23:11:44] <tomthirteen_>	 the "-e" means a query?
[23:11:55] <jynus>	 tomthirteen_, yes "execute"
[23:11:56] <tomthirteen_>	 the query is in quotes
[23:12:00] <jynus>	 you can do that
[23:12:10] <jynus>	 or just redirect the standard input, like this:
[23:12:19] <tomthirteen_>	 and instead of outfile I need ">"
[23:12:36] <jynus>	 echo "SELECT 1" | sql simplewiki_p > test.txt
[23:12:39] <tomthirteen_>	 This is standard mysql then
[23:13:09] <bd808>	 tomthirteen_: the `sql $DBNAME` part is a wrapper script we offer in /usr/bin/sql that figures out which physical database server to connect to and passes your $HOME/replica.my.cnf credentials
[23:13:10] <tomthirteen_>	 jynus can I ask you a couple more things?
[23:13:20] <jynus>	 just a couple, my bed is calling me
[23:13:31] <tomthirteen_>	 can I load an sql script and run it?
[23:13:40] <jynus>	 yes
[23:13:42] <jynus>	 that is easy
[23:14:02] <tomthirteen_>	 I have been trying to use scp and I get the same problems
[23:14:12] <jynus>	 sql simplewiki_p < script.sql
[23:14:23] <jynus>	 or:
[23:14:31] <jynus>	 sql simplewiki_p < script.sql > output.txt
[23:14:36] <jynus>	 makes sense?
[23:14:43] <tomthirteen_>	 But where is the script? I have to load the script into my directory, no?
[23:15:01] <jynus>	 that would be correct, scp to your home 
[23:15:03] <jynus>	 then run that
[23:15:18] <jynus>	 home or anywhere with db access
[23:15:27] <tomthirteen_>	 how is that command?
[23:16:09] <tomthirteen_>	 I tried something like: scp someuser@alpha.com:/somedir/somefile.txt someuser@beta.com:/anotherdir
[23:16:12] <jynus>	 you can ask me about mysql, as I was telling bd, do not know much about how tools works
[23:16:23] <jynus>	 so something long those lines
[23:16:33] <tomthirteen_>	 but it gave me a denial again
[23:16:34] <Platonides>	 tomthirteen_: you have two remotes
[23:16:42] <Platonides>	 you would need scp -3 for that
[23:16:52] <Platonides>	 but that's unlikely what you need
[23:17:06] <tomthirteen_>	 I'd like whatever is easiest
[23:17:10] <wikibugs_>	 06Labs, 10Tool-Labs: Access Denied - https://phabricator.wikimedia.org/T160585#3104443 (10MarcoAurelio) a:05Tomthirteen>03None
[23:17:14] <Platonides>	 so if you have file1.txt at yourcomputer.com
[23:17:17] <Platonides>	 you would do:
[23:17:18] <jynus>	 Platonides, I leave you with the linux cmnd lines stuff, cheers!
[23:17:36] <Platonides>	 tomthirteen_@tomcomputer.com> scp file1 tomthirteen_@labs:
[23:17:51] <Platonides>	 and that would copy it to your home at the machine "labs"
[23:17:57] <Platonides>	 buenas nochas jynus :)
[23:18:00] <Platonides>	 *noches
[23:20:05] <wikibugs_>	 06Labs, 10Tool-Labs: Access Denied - https://phabricator.wikimedia.org/T160585#3104453 (10bd808) 05Open>03Resolved a:03jcrespo Solved by @jcrespo on irc. @Tomthirteen was trying to run a query with `SELECT ... INTO OUTFILE ...` which fails because normal users do not have permission to write files on the...
[23:21:32] <tomthirteen_>	 platonides, how do I say what directory my script is in on my "home" computer
[23:22:36] <Platonides>	 tomthirteen_@tomcomputer.com> scphome/tom/folder1/file1 tomthirteen_@labs:
[23:22:44] <Platonides>	 tomthirteen_@tomcomputer.com> scp /home/tom/folder1/file1 tomthirteen_@labs:
[23:23:04] <Platonides>	 similarly, if you have a folder-scripts, in labs:
[23:23:11] <Platonides>	 tomthirteen_@tomcomputer.com> scp /home/tom/folder1/file1 tomthirteen_@labs:folder-scripts/
[23:23:39] <Platonides>	 now, you got a file output.txt in labs and want to bring it locally:
[23:23:55] <Platonides>	 tomthirteen_@tomcomputer.com> scp tomthirteen_@labs:output.txt /home/tom/folder1/
[23:25:10] <tomthirteen_>	 ok, i'm missing a piece here
[23:25:52] <Platonides>	 tell me
[23:26:47] <tomthirteen_>	 I've tried  tomthirteen_@email.arizona.com > scp /C:/scp/locked.sql tomthirteen_@labs:
[23:26:54] <tomthirteen_>	 "command not found"
[23:27:07] <Platonides>	 do you have a scp binary?
[23:27:24] <Platonides>	 what OS is your local computer?
[23:27:26] <tomthirteen_>	 I don't understand your question, so I assume no
[23:27:27] <Platonides>	 Cygwin on windows?
[23:27:30] <tomthirteen_>	 windows
[23:27:55] <Platonides>	 I mean, you need the scp binary in order to run it
[23:28:18] <Platonides>	 does ssh labs  work?
[23:28:26] <tomthirteen_>	 I don't understand your question, so I assume I don't
[23:28:54] <Platonides>	 how did you open that terminal?
[23:29:02] <tomthirteen_>	 putty
[23:29:27] <Platonides>	 but where are you writing the scp command?
[23:29:45] <tomthirteen_>	 int the shell
[23:29:48] <Platonides>	 if you are connecting from windows, WinSCP would be friendlier than scp
[23:30:01] <Platonides>	 https://winscp.net/
[23:31:00] <tomthirteen_>	 Isn't not possible to do this in the ssh in putty?
[23:35:42] <Platonides>	 tomthirteen_: you would need to use either pscp or psftp
[23:35:50] <Platonides>	 which are programs that come with putty
[23:36:51] <Platonides>	 in any case, you don't enter these commands *after* opening a putty terminal
[23:36:59] <Platonides>	 since at that point you are already at the remote side
[23:38:31] <tomthirteen_>	 ok so i need pscp or psftp
[23:38:42] <tomthirteen_>	 not scp
[23:39:07] <Platonides>	 really, you should use winscp
[23:40:17] <tomthirteen_>	 Ok, but I'll have to formally request that. I can't  install on my workstation
[23:40:54] <tomthirteen_>	 After I install it, will it be the commands you gave me before?
[23:41:16] <tomthirteen_>	 tomthirteen_@tomcomputer.com> scp /home/tom/folder1/file1 tomthirteen_@labs:
[23:41:20] <tomthirteen_>	 like that?
[23:42:47] <Platonides>	 no, after you install it you will have a window with local and remote files
[23:42:52] <Platonides>	 that you can send one way or the other
[23:42:56] <Platonides>	 it's a GUI application
[23:43:06] <tomthirteen_>	 gotcha
[23:43:16] <Platonides>	 (note there's a portable version of winscp if that help)ODs
[23:43:21] <Platonides>	 *(note there's a portable version of winscp if that helps)
[23:43:24] <tomthirteen_>	 Thank you!!! You've been immensely helpful
[23:43:55] <Platonides>	 you are welcome
[23:44:08] <tomthirteen_>	 Also, since this doesn't seem to be standard written mysql commands, what instructions should I be looking for
[23:44:21] <tomthirteen_>	 for example, it's not outfile but >
[23:46:35] <Platonides>	 well, you don't usually use outfile either
[23:46:50] <Platonides>	 whereas what jynus said will usually work everywhere
[23:48:21] <tomthirteen_>	 Ok, let me look
[23:48:37] <tomthirteen_>	 I have MariaDB installed and I can save files without problems
[23:49:11] <tomthirteen_>	 But I have been in front of my computer all day
[23:49:16] <tomthirteen_>	 I need to step away
[23:49:54] <tomthirteen_>	 Thanks again!!! You're the best