[00:07:52] <wikibugs>	 10Tool-Labs-tools-Pageviews: Add % mobile to Topviews - https://phabricator.wikimedia.org/T156081#2963591 (10MusikAnimal)
[00:21:38] <bd808>	 gry: let me find the Puppet file that provides the default config...
[00:22:12] <gry>	 hi, bd808 , i did not use puppet before, if you could guide me how to set that up it would be great
[00:29:34] <bd808>	 gry: finally found it. the config isn't in puppet anymore. It is part of the webservice python application -- https://github.com/wikimedia/operations-software-tools-webservice/blob/fca3728739087b51e074102ae8960ccc6793c37b/toollabs/webservice/services/lighttpdwebservice.py
[00:29:44] * bd808 updates that on wikitech
[00:29:53] <gry>	 bd808: good one, where do i put it ?
[00:30:27] <wikibugs>	 (03CR) 10Greg Grossmeier: [C: 04-1] "No, please keep those in -devtools. No sense in repeating everything from there into -releng. We (the relevant parties of RelEng) watch bo" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333539 (owner: 10Paladox)
[00:31:07] <bd808>	 gry: those defaults are magically put in the right place for you when you do `webservice start`. If you want to add custom overrides, take a look at https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Configuring_the_web_server
[00:31:35] <bd808>	 basically you make a file at $TOOL/.lighttpd.conf that adds to the base configuration
[00:33:14] <gry>	 bd808: 'webservice restart' does not create any lighthttpd.conf ? http://dpaste.com/3SYYY42.txt
[00:33:15] <bd808>	 I'm not sure if modstatus will work or not (never tried it on tool labs). I think that you would load the module with 'server.modules += ( "mod_status" )'
[00:34:15] <gry>	 ah ok, i will make ~/.lighttpd.conf and i think it would override the default config
[00:34:37] <bd808>	 gry:  not in the tool's directory, no. It gets written to /var/run/lighttpd/$toolname on the web exec host
[00:35:04] <gry>	 it writes «As it starts, the web server reads any configuration in $HOME/.lighttpd.conf, and merges it with the default configuration (which is likely to be adequate for most tools). »
[00:35:11] <gry>	 $HOME is /var/run/lighttpd/$toolname ?
[00:35:14] <bd808>	 correct
[00:35:44] <bd808>	 no, it would be /data/project/$toolname
[00:36:19] <gry>	 if i want custom settings for the webserver, i put it to ~/.lighttpd.conf not to /var/run/lighttpd/$toolname, right ?
[00:36:41] <bd808>	 that file is checked for and if found concatenated to the end of the base config. the result is written out only locally on the exec node that is running the tool's webserver job
[00:37:11] <bd808>	 gry: correct. put your extra stuff in  ~/.lighttpd.conf after becoming the tool
[00:37:18] <gry>	 ok :-)
[00:38:40] <gry>	 bd808: how do i append to server.modules after it was defined once ? i would like to load a new module
[00:38:57] <bd808>	 try 'server.modules += ( "mod_status" )'
[00:40:43] <gry>	 excellent, thank you bd808 :-) i got the status page load now
[00:40:54] <bd808>	 awesome
[00:41:36] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2963643 (10stwalkerster) >>! In T143349#2962892, @chasemp wrote: >> account-creation-assistance accounts-db2 > Any plans to upgrade this instance?  We've already migrated off that instan...
[04:15:10] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2963902 (10yuvipanda) I checked with @Magnus - going to shut off wikidata-wdq-mm now, can be deleted in a few days
[04:57:46] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2963953 (10TParis) I'm in the middle of a move from Hawaii to Texas, Deltaquad would be the best poc.  But if it isn't moved by March 1st then I'll take care of it.  Vr TParis
[05:03:15] <wikibugs>	 10Striker, 13Patch-For-Review: Check for 2FA protection and enforce validation of 2FA tokens - https://phabricator.wikimedia.org/T144712#2963976 (10bd808)
[05:03:19] <wikibugs>	 06Labs, 10Striker, 06Security-Team, 10Wikimedia-Site-requests, and 3 others: Add user group to wikitech granting the oathauth-api-all right - https://phabricator.wikimedia.org/T153487#2963975 (10bd808) 05Open>03Resolved
[05:10:01] <gry>	 Jamesofur: i set up https://redmine.lighttpd.net/projects/1/wiki/Docs_ModStatus and i think i found that https://tools.wmflabs.org/dupdet/compare.php?url1=https://en.wikipedia.org/index.php?title=Cognitive_peer-to-peer_networks&oldid=730517492&url2=http://www.sciencedirect.com/science/article/pii/S1084804516301084&minwords=3&minchars=13&removequotations=&removenumbers= crashes dupdet
[05:10:10] <gry>	 Jamesofur: `curl` on that sciencedirect ip also hangs
[05:11:02] <gry>	 Jamesofur: (might be only tangentially related -- i'll see whether it's enough of an issue to build up 1024+ connections and reach the limit)
[05:27:08] <gry>	 (i added connection timeout limit in a copy, will watch whose of the two ages long connections dies and whose doesn't a few hours later when i'm at the internet again)
[06:03:12] <wikibugs>	 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 4 others: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#2964119 (10bd808) Created #tool-labs-standards-committee and kicked off discussion of {T156075}
[06:04:01] <wikibugs>	 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#2964122 (10Multichill) In T143349#2963021 @yuvipanda shutdown a VM. Looks like http://wdq.wmflabs.org/stats and all queries are giving a time out now. Not sure it's related.
[06:24:37] <wikibugs>	 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#2964184 (10yuvipanda) No, I shut down the VM that was being used before wdq got its own project, so it should be unrelated.
[06:40:35] <wikibugs>	 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#2964188 (10yuvipanda) I started it back up just in case, but see no difference. Also restarted the wdq-mm service
[06:47:07] <shinken-wm>	 RECOVERY - Free space - all mounts on tools-exec-1221 is OK: OK: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found)
[07:27:41] <shinken-wm>	 PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[07:54:10] <wikibugs>	 06Tool-Labs-standards-committee: Figure out how communications and meetings will work for the Tool Labs standards committee - https://phabricator.wikimedia.org/T156075#2963376 (10zhuyifei1999) Yes, IRC channel plus mailing list. I wouldn't like to go PM-ing with 6 people at once ;)
[08:02:41] <shinken-wm>	 RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[08:24:38] <wikibugs>	 06Tool-Labs-standards-committee: Figure out how communications and meetings will work for the Tool Labs standards committee - https://phabricator.wikimedia.org/T156075#2963376 (10Ladsgroup) I'd like to have a hidden satanic cult in Alps (preferably Switzerland) for the meetings but that's up to other members. Be...
[09:06:37] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Reset password for database user p50380g50491 - https://phabricator.wikimedia.org/T155902#2964387 (10Marostegui) We granted you all on: `p50380g50491_common` the other databases didn't exist on either labs or tool boxes.
[09:11:58] <wikibugs>	 06Labs: Request creation of wikidata-federation labs project - https://phabricator.wikimedia.org/T154659#2964435 (10WMDE-leszek)
[09:23:41] <wikibugs>	 06Labs, 06Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#2964457 (10Magnus) If you can see http://wdq.wmflabs.org/ then the WDQ server is up and running. If the stats fail, that's an internal problem. I will look into it later, but won't have the...
[10:54:38] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate integration-publisher service to use a Jessie instance - https://phabricator.wikimedia.org/T156064#2964629 (10hashar) Jobs updated  INFO:jenkins_jobs.builder:Number of jobs generated:  31 INFO:jenkins_jobs.b...
[11:04:56] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate integration-publisher service to use a Jessie instance - https://phabricator.wikimedia.org/T156064#2964651 (10hashar) 05Open>03Resolved a:03hashar Validated by triggering the job operations-puppet-doc...
[11:05:00] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2964654 (10hashar)
[11:06:18] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2919816 (10hashar)
[12:31:24] <wikibugs>	 06Labs, 10DBA, 06Operations, 10netops, 13Patch-For-Review: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2964878 (10Marostegui)
[12:31:48] <wikibugs>	 06Labs, 10DBA, 06Operations, 10netops, 13Patch-For-Review: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2961118 (10Marostegui)
[12:49:35] <yuvipanda>	 !log tools purge ancient kubectl, kube-apiserver, kube-controller-manager, kube-scheduler packages from tools-k8s-master-01, these were my old terrible packages
[12:49:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:51:16] <wikibugs>	 06Labs, 10DBA, 06Operations, 10netops, 13Patch-For-Review: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2964931 (10Marostegui)
[12:51:57] <shinken-wm>	 PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[12:54:30] <yuvipanda>	 ^ is fine
[13:01:56] <shinken-wm>	 RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[13:33:31] <wikibugs>	 06Labs, 10DBA, 06Operations, 10netops, 13Patch-For-Review: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2965022 (10Marostegui)
[13:40:22] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:50:23] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[13:52:15] <yuvipanda>	 !log tools finished upgrading k8s + using debs
[13:52:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:52:25] <yuvipanda>	 !log tools upgrading k8s on worker nodes to use debs + new k8s version
[13:52:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:10:31] <wikibugs>	 06Labs, 10DBA, 06Operations, 10netops, 13Patch-For-Review: DBA plan to mitigate asw-c2-eqiad reboots - https://phabricator.wikimedia.org/T155999#2965171 (10Marostegui)
[14:10:43] <zhuyifei1999_>	 !log video pip installing pywikibot at 7ac0142 across all encoding hosts T155455
[14:10:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[14:10:47] <stashbot>	 T155455: Start using async chunked uploads - https://phabricator.wikimedia.org/T155455
[14:12:25] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2965177 (10chasemp) >>! In T143349#2962963, @chasemp wrote: > Greetings @Crazycomputers @DeltaQuad @Tparis from https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs >  > Do you have pl...
[14:13:19] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2965184 (10chasemp)
[14:15:16] <zhuyifei1999_>	 !log video git pulling v2c code to 31479a4 across all encoding hosts T155455
[14:15:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[14:19:03] <zhuyifei1999_>	 !log video depooling encoding01 & 02 and waiting for tasks to finish before restarting (reload the code) T155455
[14:19:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[14:19:07] <stashbot>	 T155455: Start using async chunked uploads - https://phabricator.wikimedia.org/T155455
[14:23:54] <ashley>	 yuvipanda: so I took a look at Andrew's most recent reminder email ([Labs-l] [Labs-announce] REMINDER: Ubuntu Precise instances on Labs will be shut down at the end of March) and it occurred to me that the one instance listed for social-tools -- social-tools1 -- is the old, broken and unused instance; http://social-tools.wmflabs.org/ uses the social-tools2 instance AFAIK. can you double-check that
[14:23:56] <ashley>	  my assumptions are correct? and even better, can you then nuke that ancient social-tools1 instance? :P
[14:25:11] <chasemp>	 ashley: can you respond on https://phabricator.wikimedia.org/T143349 instead?
[14:25:20] <chasemp>	 we are trying to contain the narrative there for everyone involved
[14:25:51] <ashley>	 chasemp: will do, thanks!
[14:26:27] <chasemp>	 ashley: thank you
[14:28:07] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2921002 (10ashley) AFAIK http://social-tools.wmflabs.org/ is using the social-tools2 instance, whereas the instance listed for deletion (social-tools1) is an older, broken, unused one. P...
[14:30:46] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2965221 (10chasemp)
[14:37:58] <yuvipanda>	 !log tools disable puppet on tools-proxy-01 (active proxy) to check deploying debianized kube-proxy on proxy-02
[14:38:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:41:00] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs by 03/31/2017 - https://phabricator.wikimedia.org/T143349#2965245 (10Krenair) >>! In T143349#2965213, @ashley wrote: > AFAIK http://social-tools.wmflabs.org/ is using the social-tools2 instance, whereas the instance listed for deletion (social-...
[14:42:45] <shinken-wm>	 PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[14:42:47] <yuvipanda>	 !log tools re-enable puppet on tools-proxy-01, test success on proxy-02
[14:42:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:44:53] <shinken-wm>	 RECOVERY - Host tools-webgrid-lighttpd-1201 is UP: PING OK - Packet loss = 0%, RTA = 1.46 ms
[14:46:10] <wikibugs>	 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#2965255 (10Andrew)
[14:46:14] <wikibugs>	 06Labs, 10DBA, 10Wikidata, 07Performance, and 3 others: Increase quota for wikidata-dev project - https://phabricator.wikimedia.org/T155042#2965251 (10Andrew) 05Open>03Resolved a:03Andrew It looks to me like there's quite a bit of quota room in wikidata-dev already.  I increased the max instance coun...
[14:52:12] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:52:44] <shinken-wm>	 RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[14:53:20] <shinken-wm>	 PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0]
[14:56:22] <wikibugs>	 06Labs, 06Operations, 10netops: asw-c2-eqiad reboots & fdb_mac_entry_mc_set() issues - https://phabricator.wikimedia.org/T155875#2965288 (10Cmjohnson) added a secondary switch, asw2-c2-eqiad.  accessible via scs port 48
[14:56:47] <wikibugs>	 06Labs, 10DBA, 06Operations, 07Tracking: Database replication problems - production and labs (tracking) - https://phabricator.wikimedia.org/T50930#2965291 (10jcrespo)
[14:56:49] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2965290 (10jcrespo) 05Open>03Resolved
[14:58:16] <zhuyifei1999_>	 !log video killing the rest of the tasks on encoding01 because they aren't going to succeed anyways; depooling encoding03; reloading encoding01 T155455
[14:58:18] <shinken-wm>	 RECOVERY - Puppet staleness on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [3600.0]
[14:58:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[14:58:19] <stashbot>	 T155455: Start using async chunked uploads - https://phabricator.wikimedia.org/T155455
[15:04:48] <hashar>	 chasemp: hello! I got rid of a Precise instance ( integration-publisher )  was quite trivial to do
[15:04:55] <hashar>	 sorry I have missed that one
[15:05:02] <wikibugs>	 06Labs, 10Analytics, 10Pageviews-API, 10wikitech.wikimedia.org: wikitech.wikimedia.org missing from pageviews API - https://phabricator.wikimedia.org/T153821#2965325 (10Milimetric) No, we should leave it open and blocked on wikitech being set up properly.  We could of course collect pageviews via some othe...
[15:05:03] <chasemp>	 hashar: no worries and thank you
[15:05:06] <hashar>	 the other Precise instances for CI are still for PHP 5.3 :(
[15:05:24] <hashar>	 I did try to compile Zend 5.3 against Jessie but without success. My C is next to nill!
[15:06:54] <chasemp>	 hashar: yeah I was going to ask for a bit more info there so that we are all on teh same page, that will possibly end up being an outlier and keep the "goal" yellow
[15:07:04] <chasemp>	 which if it has to be then ok but wanted to walk through it and make tickets and such
[15:07:08] <chasemp>	 but probably later this week
[15:07:35] <hashar>	 the end idea is to still be able to run tests for MediaWiki 1.23 which is Zend 5.3.
[15:08:00] <hashar>	 maybe we would be fine with a single instance. Or maybe we can look at running them manually in a Vagrant Precise box instead of via ci
[15:08:09] <hashar>	 this way the goal is greenish and Precise gone by end of March :]
[15:08:35] <yuvipanda>	 ^ would be great :D
[15:08:36] <yuvipanda>	 or in a precise docker / rkt container ;)
[15:09:21] <hashar>	 yeah 
[15:09:36] <hashar>	 though I have doubt we will have anything running on rkt by end of quarter!
[15:12:55] <yuvipanda>	 hashar: the advantage of rkt over docker is that you can easily do 'sudo rkt run --image=<something> --volumes=<something> <path>' and it'll behave exactly like just running <path>
[15:12:59] <yuvipanda>	 unlike docker which won't
[15:13:07] <hashar>	 yuvipanda: yeah totally agreed
[15:13:21] <hashar>	 I think the misconception is that we have experimented with plain docker + Dockerfile
[15:13:36] <hashar>	 and really dont care whether it is Docker itself or Rkt running the containers
[15:13:37] <hashar>	 well
[15:13:42] <hashar>	 actually no we care
[15:13:48] <hashar>	 we want to use whatever ops uses :]
[15:14:13] <yuvipanda>	 hashar: yah, but this would be completely unrelated to all of that :) it'll just be a small hack that'll work with the existing infrastructure to allow us to get rid of precise
[15:14:45] <hashar>	 yup
[15:14:59] <hashar>	 maybe that could be a first good poc to move CI to k8s
[15:15:14] <hashar>	 then, we have no clue where to host the CI containers yet :/
[15:15:20] <yuvipanda>	 no :)
[15:15:28] <yuvipanda>	 it's all unrelated, and you'll just be able to replace /usr/bin/php with a different script
[15:15:55] <hashar>	 ??????????????????????????????
[15:16:03] <hashar>	 hold on
[15:16:15] <yuvipanda>	 that's why I was talking about rkt.
[15:16:25] <hashar>	 just have to install rkt  then it can run the container without the need of some daemon / etcd / k8s etc?
[15:16:42] <yuvipanda>	 this is purely a way to run php 5.3 on trusty by just putting it in a container image and running it with rkt
[15:16:42] <yuvipanda>	 hashar: exactly
[15:16:43] <yuvipanda>	 hashar: rkt has no daemons
[15:17:25] <hashar>	 so we would craft a Precise docker image that has everything we need.  Make it so the entry point is /usr/bin/php
[15:17:30] <hashar>	 then  instead of  php foo
[15:17:53] <hashar>	 run rkt --image=ci-precise tests/phpunit ? :D
[15:18:28] <yuvipanda>	 hashar: exactly
[15:18:42] <yuvipanda>	 hashar: and mount whatever you want from the host
[15:18:55] <yuvipanda>	 hashar: so if you want /srv/home/something on the host to be same on container, that's trivial
[15:19:26] <hashar>	 and rkt runs the container inside a kvm right?
[15:20:07] <yuvipanda>	 hashar: not by default no
[15:20:14] <yuvipanda>	 hashar: that won't really be useful for us since we're already running inside kvm
[15:20:30] <hashar>	 what I am wondering is whether we could rkt the containers on the prod machine contint1001 / contint2001
[15:20:45] <hashar>	 which together probably have more power combined than all the CI labs instances
[15:21:25] <yuvipanda>	 hashar: > sudo -E rkt run --volume curdir,kind=host,source=$PWD --volume passwd,kind=host,source=/etc/passwd --volume=group,kind=host,source=/etc/group --volume=resolv,kind=host,source=/etc/resolv.conf --insecure-options=image --interactive docker://ubuntu:latest --mount volume=passwd,target=/etc/passwd --mount volume=group,target=/etc/group --mount volume=resolv,target=/etc/resolv.conf --mount volume=curdir,target=$PWD --user=`id -u`
[15:21:26] <yuvipanda>	 --group=`id -g` --net=host --inherit-env --hostname=`hostname` --set-env=USER=$USER --exec /bin/bash -- -c 'cat /etc/passwd'
[15:21:47] <yuvipanda>	 that will run whatever you pass affter -- as the same user as the one executing command, but inside a docker container
[15:22:05] <yuvipanda>	 hashar: I'd rather have this run in a labs instance, since running things in prod will be more work
[15:22:18] <yuvipanda>	 hashar: the smallest operation would be to just change the entrypoint php5.3 uses to use a wrapper bash script that runs it under rkt
[15:24:02] <hashar>	 okkkkk
[15:24:38] <hashar>	 the fun will be in provisioning an image with the proper deps
[15:24:56] <hashar>	 then I guess if we do that for Precise, that will serve for other jobs / Jessie
[15:25:09] <yuvipanda>	 hashar: should be fairly trivial :)
[15:29:58] <hashar>	 yuvipanda: I filled a task yesterday about migrating to bootstrapvz
[15:30:12] <hashar>	 I went with a different image building system (  diskimage-builder from openstack )
[15:30:27] <yuvipanda>	 hashar: migrating what?
[15:30:28] <yuvipanda>	 hashar: bootstrapvz only supports debian I think
[15:31:19] <hashar>	 o
[15:31:28] <hashar>	 I was talking about the CI images uploaedd to openstack that Nodepool uses
[15:31:41] <hashar>	 the aim being to phase out trusty entirely from CI
[15:31:44] <hashar>	 and solely run on Jessie
[15:32:28] <zhuyifei1999_>	 !log video git pulling v2c code to b2d6e84 across all encoding hosts T155455
[15:32:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[15:32:31] <stashbot>	 T155455: Start using async chunked uploads - https://phabricator.wikimedia.org/T155455
[15:32:58] <yuvipanda>	 hashar: no I mean, keep using the jessie / trusty images you are currently using
[15:33:15] <yuvipanda>	 hashar: and run php5.3 from inside them
[15:33:16] <yuvipanda>	 very minimal change
[15:36:24] <shinken-wm>	 PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[15:38:19] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Reset password for database user p50380g50491 - https://phabricator.wikimedia.org/T155902#2965491 (10Tb) Apologies, double-underscores seem to get eaten by Phabricator's markup parser.    On s1.labsdb: ``` p50380g50491_common p50380g50491__rlrl_enwiki_p p50380g50491__rlrl_ptwiki_p...
[15:41:20] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[15:43:10] <shinken-wm>	 PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[15:45:38] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Reset password for database user p50380g50491 - https://phabricator.wikimedia.org/T155902#2965532 (10Marostegui) Thanks for the clarification.  I have granted access to those databases. Please check them and let us know if that works!
[15:49:28] <yuvipanda>	 !log tools  clush -g all 'sudo rm /usr/local/bin/kube*' to get rid of old kube related binaries
[15:49:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:04:15] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Reset password for database user p50380g50491 - https://phabricator.wikimedia.org/T155902#2958263 (10jcrespo) @Marostegui and others ops, grants are wildcards, **never** use _ without escaping (\_) on a grant. It is not a big deal here, but it can lead to security problems.
[16:04:25] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Add passwords::redis::ores_password [labs/private] - 10https://gerrit.wikimedia.org/r/333924
[16:13:10] <shinken-wm>	 RECOVERY - Puppet run on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:16:04] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] Add passwords::redis::ores_password [labs/private] - 10https://gerrit.wikimedia.org/r/333924 (owner: 10Alexandros Kosiaris)
[16:16:31] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Reset password for database user p50380g50491 - https://phabricator.wikimedia.org/T155902#2965610 (10jcrespo) @Tb your grants have been added- you should be able to access old data- however, you should consider those grants temporary, until you rename the databases to start with `...
[16:17:25] <zhuyifei1999_>	 !log video encoding03 free disk space < 10G, renicing PID 9136 (50c81becdbd5a1c5) & 3499 (a32b3fa5dc0e2123) & 3611 (0e73d27e7cb00284) to niceness of 1, so these near-complete tasks can finish early and clear some disk space
[16:17:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[16:21:07] <shinken-wm>	 PROBLEM - SSH on tools-webgrid-lighttpd-1201 is CRITICAL: Connection refused
[16:23:25] <shinken-wm>	 PROBLEM - Host tools-webgrid-lighttpd-1201 is DOWN: CRITICAL - Host Unreachable (10.68.18.45)
[16:26:31] <shinken-wm>	 RECOVERY - Host tools-webgrid-lighttpd-1201 is UP: PING OK - Packet loss = 0%, RTA = 476.48 ms
[16:28:58] <wikibugs>	 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#2965667 (10Andrew)
[16:29:00] <wikibugs>	 06Labs: Increase resource quota for dwl - https://phabricator.wikimedia.org/T152456#2965665 (10Andrew) 05Open>03stalled Sorry for the delay!  I've adjusted quotas so that you have headroom to add one bigram instance on top of what you have in the project now.  After you're done migrating and deleting old ins...
[16:29:21] <wikibugs>	 06Labs: Lower quotas to current usage for dwl (when ready) - https://phabricator.wikimedia.org/T152456#2965668 (10Andrew)
[16:31:08] <shinken-wm>	 RECOVERY - SSH on tools-webgrid-lighttpd-1201 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0)
[16:34:16] <shinken-wm>	 PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [43200.0]
[16:38:37] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2965716 (10Superyetkin) Could you please share the configuration details of new servers? Most of my tools [[http://tools.wmflabs.org/superyetkin/kategorisizsayfalar.php | like this]] (running on trwiki...
[16:39:19] <shinken-wm>	 RECOVERY - Puppet staleness on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [3600.0]
[16:44:23] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: enwiki_p replica on s1 is corrupted - https://phabricator.wikimedia.org/T134203#2965719 (10jcrespo) @Superyetkin I cannot guarantee it will not change in the future, but you can connect, in the case of **enwiki** to the `labsdb-web.eqiad.wmnet` host for short-lived, web-like reque...
[16:52:41] <wikibugs>	 06Labs, 10DBA, 10Wikidata, 07Performance, and 3 others: Increase quota for wikidata-dev project - https://phabricator.wikimedia.org/T155042#2965728 (10Ladsgroup) We cleaned some instances and it's okay now. Probably we will make more soon.
[16:58:59] <shinken-wm>	 PROBLEM - Host tools-mail-01 is DOWN: CRITICAL - Host Unreachable (10.68.22.188)
[17:02:19] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Joal was modified, changed by Joal link https://wikitech.wikimedia.org/w/index.php?diff=1383744 edit summary: 
[17:08:09] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-exec-1221 is CRITICAL: CRITICAL: tools.tools-exec-1221.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-exec-1221.diskspace.root.byte_percentfree (<44.44%)
[17:15:35] <andrewbogott>	 !log tools stopping tools-mail, backing up, upgrading from precise to trusty
[17:15:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:17:05] <andrewbogott>	 !log tools backing up tools-mail to ~root/8c499e6e-1b79-4bb1-8f7f-72fee1f74ea5-backup on labvirt1009
[17:17:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:18:08] <shinken-wm>	 PROBLEM - Host tools-mail is DOWN: CRITICAL - Host Unreachable (10.68.16.27)
[17:19:02] <andrewbogott>	 !log tools restarting tools-mail, beginning do-release-upgrade -d -q
[17:19:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:22:25] <shinken-wm>	 RECOVERY - Host tools-mail is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms
[17:25:19] <shinken-wm>	 PROBLEM - Puppet run on tools-mail is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[17:46:01] <shinken-wm>	 RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[17:51:31] <andrewbogott>	 !log tools rebooting tools-mail post upgrade
[17:51:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:52:19] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Reset password for database user p50380g50491 - https://phabricator.wikimedia.org/T155902#2965899 (10Tb) Great thanks. although I missed one in the list above;  can you grant all to s51111 on p50380g50491_inconsistent_redirects on s1.labsdb also please.
[18:00:40] <yuvipanda>	 andrewbogott: I just did a test of the tools-mail functionality and it seems ok :D
[18:00:56] <andrewbogott>	 !log tools apt-get autoremove on tools-mail
[18:00:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:01:08] <andrewbogott>	 yuvipanda: great, I'm going to do a bit more cleanup and reboot again before declaring victory
[18:01:21] <yuvipanda>	 andrewbogott: awesome, ok
[18:05:21] <shinken-wm>	 RECOVERY - Puppet run on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0]
[18:12:02] <andrewbogott>	 yuvipanda: there are like 25 old kernels on that box, apt is still cleaning
[18:12:12] <yuvipanda>	 andrewbogott: :D nice
[18:12:20] <yuvipanda>	 andrewbogott: we should clush a clean on all boxes sometime
[18:12:38] <andrewbogott>	 yeah, wouldn't hurt
[18:13:02] <andrewbogott>	 the kernel post-uninstall script seems to do a system-wide 'find' or something, it's super slow
[18:13:58] <andrewbogott>	 !log one last reboot of tools-mail
[18:13:58] <stashbot>	 Unknown project "one"
[18:14:03] <andrewbogott>	 !log tools one last reboot of tools-mail
[18:14:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:23:49] <zhuyifei1999_>	 !log video killed the rest of tasks on encoding02 & encoding03, they are gonna fail anyways
[18:23:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[18:27:53] <zhuyifei1999_>	 !log video repooling encoding02 & encoding03, depooling encoding01 T155455
[18:27:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[18:27:56] <stashbot>	 T155455: Start using async chunked uploads - https://phabricator.wikimedia.org/T155455
[18:37:03] <shinken-wm>	 PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[18:40:47] <wikibugs>	 06Labs: Clean up backups of tools-mail on labvirt1009 - https://phabricator.wikimedia.org/T156160#2966033 (10Andrew)
[18:43:44] <paladox>	 !log phabricator deleting phab-03 and phab-05 to allow us to create one large instance for replacement for phab-01 (might)
[18:43:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL
[18:43:50] <mutante>	 !log phabricator please stop using phab-01,-03,-05, none of them use puppet. please use instance "phabricator", this actually has the prod role
[18:43:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL
[18:45:18] <paladox>	 !log phabricator deleting phabricator instance and recreating it, replacing phab-01 with a phabricator instance :)
[18:45:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL
[19:06:47] <musikanimal>	 hello! so it appears the crontab for all of my tools has been cleared!?
[19:07:43] <musikanimal>	 yuvipanda bd808 ? was this part of the NFS maintenance? I seem to recall the jobs running fine after that happened
[19:08:54] <joal>	 Hi labs people :) Do you think someone could take care of that one before tomorrow (it would unblock me for tomorrow morning): https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Joal
[19:12:34] <musikanimal>	 ^ madhuvishy, about the crontabs. Any ideas? :'(
[19:12:57] <madhuvishy>	 musikanimal: no that wasn't part of the maintenance
[19:13:42] <musikanimal>	 well I wonder what happened. I assume this was not intentional
[19:14:12] <madhuvishy>	 no, tools was unaffected in the maintenance
[19:15:31] <musikanimal>	 cool, well looking at my bot's contribs, I'm going to guess the crontabs were cleared some time in the past 8 hours
[19:15:42] <musikanimal>	 which is not cool :'(
[19:15:53] <madhuvishy>	 oh - which bot is this?
[19:15:55] <musikanimal>	 for some of the tools I don't have the configurations backed up
[19:15:56] <madhuvishy>	 tool*
[19:15:58] <musikanimal>	 all of them
[19:16:06] <musikanimal>	 musikbot, eranbot, xtools
[19:16:10] <madhuvishy>	 looking
[19:16:23] <musikanimal>	 all of the crontabs are in the initial state, as if it were a fresh new tool
[19:16:33] <madhuvishy>	 that's.. weird
[19:18:13] <musikanimal>	 I know you can run sudo something something to see when the crontabs were last edited, and perhaps by whom
[19:19:02] <musikanimal>	 sudo ls -l /var/spool/cron/crontabs/$USER
[19:19:19] <zhuyifei1999_>	 tools.yifeibot@tools-bastion-02:~$ ssh tools-cron-01
[19:19:20] <zhuyifei1999_>	 Connection closed by 10.68.23.89
[19:20:35] <zhuyifei1999_>	 tools-cron-01 malfunction?
[19:21:07] <musikanimal>	 I hope so! that is better than them being completely cleared, assuming we can get it back up and running
[19:21:12] <joal>	 actually labs team, sorry to have bothered, looks like I was on the wrong path - nevermind my previous request
[19:22:26] <musikanimal>	 zhuyifei1999_: I was able to SSH in using my normal bastion user, without "become"ing a tool
[19:23:39] <zhuyifei1999_>	 hmm
[19:24:17] <madhuvishy>	 i can login to tools-cron-01
[19:24:21] <zhuyifei1999_>	 musikanimal: time to back up your crontab :P
[19:24:43] <musikanimal>	 *was* time, too late now :(
[19:25:08] <zhuyifei1999_>	 you can still `become` in tools-cron-01
[19:25:39] <musikanimal>	 oh nice!!!
[19:25:43] <musikanimal>	 okay, backing up now!
[19:26:03] <zhuyifei1999_>	 basically all your crontabs are saved/edited/executed on tools-cron-01
[19:26:56] <zhuyifei1999_>	 idk why ssh within service accounts are broken
[19:27:27] <madhuvishy>	 musikanimal: yeah i can see all the crons are still intact there
[19:27:32] <madhuvishy>	 but something is broken
[19:28:06] <zhuyifei1999_>	 madhuvishy: is there a debug log for sshd?
[19:28:15] <zhuyifei1999_>	 it doesn't seem like an auth failure
[19:29:01] <zhuyifei1999_>	 last lines with ssh -vvv:
[19:29:10] <zhuyifei1999_>	 debug2: userauth_hostbased: chost tools-bastion-02.tools.eqiad.wmflabs.
[19:29:10] <zhuyifei1999_>	 debug3: ssh_msg_send: type 2
[19:29:11] <zhuyifei1999_>	 debug1: permanently_drop_suid: 51201
[19:29:11] <zhuyifei1999_>	 debug3: ssh_msg_recv entering
[19:29:11] <zhuyifei1999_>	 debug3: ssh_keysign: [child] pid=23881, exec /usr/lib/openssh/ssh-keysign
[19:29:12] <zhuyifei1999_>	 debug2: we sent a hostbased packet, wait for reply
[19:29:12] <zhuyifei1999_>	 Connection closed by 10.68.23.89
[19:36:05] <madhuvishy>	 musikanimal: i'm still not sure what's going on, but fyi we also backup crontabs in nfs (TIL) 
[19:36:24] <musikanimal>	 good to know :)
[19:37:07] <musikanimal>	 I've just now backed all of mine up by SSH in to tools-cron-01
[19:37:39] <madhuvishy>	 https://www.irccloud.com/pastebin/beHb0Adc/
[19:40:03] <zhuyifei1999_>	 I think I'll just file a ticket
[19:41:54] <madhuvishy>	 zhuyifei1999_: thanks
[19:42:07] <musikanimal>	 also I forgot to change back my timezone on wiki, the last edit via cron by my bot was 40 minutes ago
[19:42:18] <musikanimal>	 so everything is probably still working
[19:42:22] <madhuvishy>	 musikanimal: yeah it's all working
[19:42:38] <musikanimal>	 yay :)
[19:42:40] <madhuvishy>	 not sure why your local crontab appears cleared
[19:43:01] <musikanimal>	 yeah, this is true for all the tools that I have access to
[19:43:16] <musikanimal>	 I guess if I need to modify them I'll have to go through tools-cron-01?
[19:46:36] <chasemp>	 don't modify crons directy on the tools-cron hosts
[19:46:39] <madhuvishy>	 musikanimal: ideally not
[19:46:48] <musikanimal>	 ok
[19:47:00] <madhuvishy>	 we're looking right now
[19:47:10] <wikibugs>	 06Labs, 10Tool-Labs: Tool labs crontab host tools-cron-01 cannot be ssh-ed into within a service user - https://phabricator.wikimedia.org/T156168#2966289 (10zhuyifei1999)
[19:54:11] <framawiki>	 Hi all, is it normal that I can't access to cron on tools ?
[19:54:12] <framawiki>	 tools.framabot@tools-bastion-03:~$ crontab -l
[19:54:13] <framawiki>	 Connection closed by 10.68.23.89
[19:54:46] <madhuvishy>	 framawiki: we are currently looking into it
[19:54:52] <zhuyifei1999_>	 framawiki: T156168 just filed
[19:54:52] <stashbot>	 T156168: Tool labs crontab host tools-cron-01 cannot be ssh-ed into within a service user - https://phabricator.wikimedia.org/T156168
[19:57:06] <framawiki>	 Thanks for the link, do you know if crons are normally executed ?
[19:57:39] <framawiki>	 "Crontabs are intact."
[19:58:53] <zhuyifei1999_>	 framawiki: yes
[19:59:34] <chasemp>	 musikanimal: zhuyifei1999_ does crontab -l as a tool user normally work from teh bastion? (just verifying)
[19:59:46] <zhuyifei1999_>	 yes
[20:00:00] <chasemp>	 k
[20:04:19] <andrewbogott>	 joal: you appear to already be a member of the tools project; is there anything else you need?
[20:04:29] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Joal was modified, changed by Andrew Bogott link https://wikitech.wikimedia.org/w/index.php?diff=1385324 edit summary: 
[20:08:25] <joal>	 Hi andrewbogott, sorry, made a mistake in procdure: I wanted to create a new tool
[20:09:21] <joal>	 andrewbogott: I however have another question - I created the tool sqoop-tool, but it seems I don't have the replica.cnf file allowing db access :(
[20:10:12] <andrewbogott>	 joal: ok, so, just to confirm… you already can/know how to create tools, correct?
[20:10:34] <joal>	 andrewbogott: I followed the doc best I could :)
[20:10:43] <andrewbogott>	 I don't know how replica.cnf works at all, so I defer to madhuvishy and/or bd808 
[20:10:58] <joal>	 I now can become sqoop-tool on tools-bastion-03
[20:11:02] <bd808>	 joal: wait 10 minutes
[20:11:07] <joal>	 sure bd808 
[20:11:08] <madhuvishy>	 joal: i'll look and fix in a bit
[20:11:19] <bd808>	 it should be faster than that but sometimes it's not
[20:11:21] <madhuvishy>	 if it doesn't show up
[20:11:27] <joal>	 wow, 3 people just me :)
[20:11:30] <joal>	 Thanks guys !
[20:11:53] <joal>	 bd808: I created the tool like 2 or 3 hours ago
[20:15:01] <chasemp>	 joal: the system that does that is currently frozen for puppet and needs some attention but I'm stuck elsewhere
[20:15:07] <chasemp>	 if not there by tomorrow drop me a note
[20:15:38] <joal>	 chasemp, madhuvishy, bd808, andrewbogott: Thank you all for the help, will check tomorrow and let you know !
[20:41:07] <madhuvishy>	 fix coming in for crontab editing issue - should be resolved soon
[20:42:10] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 032] Add link to OATH documentation on wikitech [labs/striker] - 10https://gerrit.wikimedia.org/r/333321 (owner: 10BryanDavis)
[20:43:43] <wikibugs>	 (03Merged) 10jenkins-bot: Add link to OATH documentation on wikitech [labs/striker] - 10https://gerrit.wikimedia.org/r/333321 (owner: 10BryanDavis)
[20:43:50] <wikibugs>	 06Labs, 10Tool-Labs: Rewrite /usr/local/bin/crontab in python; fix bugs - https://phabricator.wikimedia.org/T156174#2966501 (10bd808)
[20:59:37] <wikibugs>	 06Labs, 10Tool-Labs, 13Patch-For-Review: Tool labs crontab host tools-cron-01 cannot be ssh-ed into within a service user - https://phabricator.wikimedia.org/T156168#2966289 (10chasemp) seems to work now from bastion-03 to tools-cron-01 using `crontab -e` as a test tool.  Thanks for the report @zhuyifei1999...
[20:59:43] <wikibugs>	 06Labs, 10Tool-Labs, 13Patch-For-Review: Tool labs crontab host tools-cron-01 cannot be ssh-ed into within a service user - https://phabricator.wikimedia.org/T156168#2966594 (10chasemp) 05Open>03Resolved a:03chasemp
[21:02:11] <wikibugs>	 10Tool-Labs-tools-LTA-Knowledgebase: Add option for tool admins to change write access - https://phabricator.wikimedia.org/T156181#2966635 (10DatGuy)
[21:02:20] <yuvipanda>	 joal: you should have a replica.my.cnf now
[21:30:24] <zhuyifei1999_>	 !log video git pulling v2c code to 8cf788e across all encoding hosts
[21:30:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[21:33:04] <zhuyifei1999_>	 !log video depooling encoding02 & 03, repooling encoding01
[21:33:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[21:37:00] <wikibugs>	 06Labs, 10Horizon, 06Operations, 07Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#2966807 (10Andrew) - I will double-check the caching, although I'm pretty sure I verified that the cache was working previously.  - I'm currently experimenting with the next rev of Hor...
[21:52:39] <wikibugs>	 06Labs, 10Tool-Labs, 13Patch-For-Review: Tool labs crontab host tools-cron-01 cannot be ssh-ed into within a service user - https://phabricator.wikimedia.org/T156168#2966858 (10MusikAnimal) Thanks for the quick fix!
[22:01:18] <wikibugs>	 (03PS1) 10BryanDavis: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/333989
[22:02:08] <wikibugs>	 (03CR) 10BryanDavis: [C: 032] Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/333989 (owner: 10BryanDavis)
[22:02:15] <wikibugs>	 (03Merged) 10jenkins-bot: Bump striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/333989 (owner: 10BryanDavis)
[22:09:42] <wikibugs>	 10Striker, 15User-bd808: Deploy Striker account creation and management workflow - https://phabricator.wikimedia.org/T156195#2966952 (10bd808)
[22:10:20] <wikibugs>	 10Striker, 15User-bd808: Deploy Striker account creation and management workflow - https://phabricator.wikimedia.org/T156195#2966972 (10bd808)
[22:25:33] <joal>	 Awesome ! Thanks yuvipanda :)
[22:48:58] <wikibugs>	 10Striker, 15User-bd808: Deploy Striker account creation and management workflow - https://phabricator.wikimedia.org/T156195#2967121 (10bd808)
[23:02:33] <wikibugs>	 06Labs, 10Tool-Labs: Rewrite /usr/local/bin/crontab in python; fix bugs - https://phabricator.wikimedia.org/T156174#2966501 (10scfc) It would be nice to split out the "template" bit of it; i. e., let Puppet write `@cron_host` to some file in `/etc` and make `/usr/local/bin/crontab` read from that instead of ch...
[23:50:38] <wikibugs>	 10Striker, 15User-bd808: Deploy Striker account creation and management workflow - https://phabricator.wikimedia.org/T156195#2967338 (10bd808) The prod secrets need be appended to the existing to the existing `striker::uwsgi::secret_config` hiera hash and look something like: ``` striker::uwsgi::secret_config:...