[00:03:39] <sitic>	 yay, thanks :-)
[00:04:07] * sitic can stop running his very small private redis as grid job
[00:05:28] <YuviPanda>	 tch tch :)
[00:40:46] <T13|mobile>	 Does anyone know if http://toolserver.org/~merphant/cgi-bin/wikiholic.cgi got ported over or if the source is available for me to fork?
[01:07:57] * T13|mobile takes that as a no and emails the creator who hasn't been active in half a decade on the slum chance.
[01:08:13] <T13|mobile>	 %s/slum/slim/g
[01:57:54] <T13|mobile>	 YuviPanda: how do I set up cgi on toollabs for a .pl script I inherited?
[01:58:08] <YuviPanda>	 ah, good question. I’m not sure at all.
[01:58:19] <YuviPanda>	 just put it in public_html and try tools.wmflabs.org/<toolname>/<filename>.pl
[02:00:24] <T13|mobile>	 Tools.wmflabs.org/technical-13/<filename>.pl. okay will try.  Will I need to start a webservice in my space and should i use webservice or webservice2 (what's the diif)?
[02:01:15] <YuviPanda>	 they’re the same. webservice2 defaults to trusty, webservice defaults to boring ol’ precise.
[02:01:24] <YuviPanda>	 also you should create a separate tool for it rather than put it under your own name
[02:01:48] <T13|mobile>	 How do I create a tool?
[02:01:59] <T13|mobile>	 Yes, I'm a newb
[02:02:20] <YuviPanda>	 T13|mobile: go to tools.wmflabs.org
[02:02:31] <YuviPanda>	 T13|mobile: there’s a ‘create new tool’ under userful links
[02:03:26] <T13|mobile>	 Found it
[02:15:46] <T13|mobile>	 What about a C program? Do I have to gcc it?
[02:15:51] <T13|mobile>	 YuviPanda: ^
[02:16:25] <YuviPanda>	 as a web application?
[02:16:48] <T13|mobile>	 Yes, it's the old wiki2html that use to be on tools
[02:16:58] <YuviPanda>	 oh, I guess...
[02:17:07] <YuviPanda>	 you’ll have to use portgrabber for it
[02:17:13] <YuviPanda>	 let me find it
[02:18:05] <YuviPanda>	 T13|mobile: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web#Other_web_servers
[02:21:56] <grrrit-wm>	 (03PS1) 10Legoktm: Only match MediaWiki-extensions-CiteThisPage once [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/205499 
[02:22:13] <legoktm>	 YuviPanda: ^ 
[02:22:40] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] Only match MediaWiki-extensions-CiteThisPage once [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/205499 (owner: 10Legoktm)
[02:22:43] <grrrit-wm>	 (03Merged) 10jenkins-bot: Only match MediaWiki-extensions-CiteThisPage once [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/205499 (owner: 10Legoktm)
[02:23:27] <legoktm>	 ty
[02:26:27] <T13|mobile>	 I uploaded the files, will finish set up later.
[02:26:36] <T13|mobile>	 Too sleepy.
[02:26:57] <T13|mobile>	 Bookmarked that page though.
[03:39:01] <ircnotifier>	 !log tools.wikibugs legoktm: Deployed 8e88fc89deaa41b2a720845f5d20aa871ffa09d9 Add Blueprint skin to notify list for #wikimedia-design wb2-phab
[03:39:04] <labs-morebots>	 Logged the message, Master
[03:39:08] <ircnotifier>	 !log tools.wikibugs legoktm: Deployed 8e88fc89deaa41b2a720845f5d20aa871ffa09d9 Add Blueprint skin to notify list for #wikimedia-design wb2-irc
[03:39:10] <labs-morebots>	 Logged the message, Master
[03:53:14] <YuviPanda>	 shinken-wm: are you around?
[03:53:17] <YuviPanda>	 hmm
[04:31:56] <GerardM->	 good morning
[04:32:12] <GerardM->	 are there issues ? Autolist http://tools.wmflabs.org/autolist/index.php does not load
[04:32:18] <harej>	 hoi
[04:32:24] <GerardM->	 now it does ...
[04:37:38] <GerardM->	 thanks
[04:38:03] <harej>	 you're welcome
[04:50:57] <wikibugs>	 7Tool-Labs: s7.labsdb long lag - https://phabricator.wikimedia.org/T96646#1223281 (10eranroz) 3NEW a:3coren
[05:36:13] <wm-bot>	 Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by Tbayer (WMF) link https://www.mediawiki.org/w/index.php?diff=1560051 edit summary: [+230] +quarterly review; some WP:BOLD updates, removing one outdated todo item of many
[05:36:50] <YuviPanda>	 HaeB: wheeeee
[05:36:52] <YuviPanda>	 Ty
[05:37:11] <HaeB>	 no worries, enough left to do ;)
[05:37:31] <YuviPanda>	 Heh isn't that always true :l
[05:37:32] <YuviPanda>	 :)
[08:08:51] <_joe_>	 !lgo deployment-prep installing HHVM 3.6 and the corresponding extensions on deployment-mediawiki01
[08:08:58] <_joe_>	 !log deployment-prep installing HHVM 3.6 and the corresponding extensions on deployment-mediawiki01
[08:09:02] <labs-morebots>	 Logged the message, Master
[08:33:48] <_joe_>	 !log deployment-prep rollback installation of hhvm 3.6
[08:33:51] <labs-morebots>	 Logged the message, Master
[08:54:08] <wikibugs>	 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 5Continuous-Integration-Isolation: OpenStack API account to control `contintcloud` labs project - https://phabricator.wikimedia.org/T86170#1223677 (10hashar) It works now, `nodepoolmanager` still had a temporary password on wikitech. I have...
[09:11:53] <wikibugs>	 7Tool-Labs: Multiple queue runners on tools-mail - https://phabricator.wikimedia.org/T74867#1223735 (10valhallasw) The times are also odd: ``` 2015-04-14 22:29:05 2015-04-17 03:09:07 2015-04-18 14:49:04 2015-04-19 13:49:04 2015-04-20 14:49:09 ```  Always a few seconds after the 9th minute. I thought about cron,...
[09:31:19] <wikibugs>	 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 5Continuous-Integration-Isolation: OpenStack API account to control `contintcloud` labs project - https://phabricator.wikimedia.org/T86170#1223794 (10hashar) I have adjusted the nodepool setting file and it managed to creates its first ever...
[09:42:17] <wikibugs>	 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 5Continuous-Integration-Isolation, 7Nodepool: OpenStack API account to control `contintcloud` labs project - https://phabricator.wikimedia.org/T86170#1223837 (10hashar)
[09:42:27] <wikibugs>	 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 5Continuous-Integration-Isolation, 7Nodepool: OpenStack API account to control `contintcloud` labs project - https://phabricator.wikimedia.org/T86170#1223838 (10hashar) a:3hashar
[10:52:37] <jzerebecki>	 YuviPanda, Coren: https://tools.wmflabs.org/ does not have links anymore even for webservices that are running
[10:53:10] <jzerebecki>	 btw. does the new service manifest thingie send mails when it restarts stuff?
[11:59:16] <gifti>	 coren: if you want to, you can reboot tools-exec-gift
[13:37:30] <wikibugs>	 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 5Continuous-Integration-Isolation, 7Nodepool: OpenStack API account to control `contintcloud` labs project - https://phabricator.wikimedia.org/T86170#1224349 (10hashar) 5Open>3Resolved Nodepool can access the OpenStack API just fine no...
[13:49:53] <wikibugs>	 6Labs: Move ldap host-record creation out of OpenStackManager and into sink - https://phabricator.wikimedia.org/T96677#1224373 (10Andrew)
[13:52:44] <wikibugs>	 6Labs, 10Beta-Cluster: Migrate deployment-prep to new labvirt hosts - https://phabricator.wikimedia.org/T96678#1224378 (10Andrew) 3NEW a:3Andrew
[14:00:19] <wikibugs>	 6Labs, 10Beta-Cluster: Migrate deployment-prep to new labvirt hosts - https://phabricator.wikimedia.org/T96678#1224401 (10hashar) I guess it is fine. There will be surely some side effects on the beta cluster but if we announce it to the engineering and qa lists people would know it is going to be flapping for...
[14:05:07] <wikibugs>	 6Labs, 10Beta-Cluster: Migrate deployment-prep to new labvirt hosts - https://phabricator.wikimedia.org/T96678#1224437 (10Andrew) I've done about 20 instances with no corruption -- the worst case is that the instance just doesn't copy.   I can backup that instance but I'd need to halt it first -- would you lik...
[15:01:13] <wikibugs>	 6Labs, 10Beta-Cluster: Migrate deployment-prep to new labvirt hosts - https://phabricator.wikimedia.org/T96678#1224548 (10greg) FYI @mmodell, @dduvall, @thcipriani
[16:02:12] <YuviPanda>	 jzerebecki: doesn't send emails. Probably should.
[16:13:08] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/B was created, changed by B link https://wikitech.wikimedia.org/wiki/Nova+Resource%3aTools%2fAccess+Request%2fB edit summary: Created page with "{{Tools Access Request |Justification=Various tasks having to do with images on enwiki |Completed=false |User Name=B }}"
[16:28:26] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Dgdiego was created, changed by Dgdiego link https://wikitech.wikimedia.org/wiki/Nova+Resource%3aTools%2fAccess+Request%2fDgdiego edit summary: Created page with "{{Tools Access Request |Justification=I plan to use the Tools project for research about Wikipedia |Completed=false |User Name=Dgdiego }}"
[16:38:54] <andrewbogott>	 Krinkle: cvn project migrating now
[16:43:17] <wikibugs>	 6Labs, 10Continuous-Integration: Create an instance image like m1.small with 2 CPUs and 30GB space - https://phabricator.wikimedia.org/T96706#1224899 (10Krinkle) 3NEW a:3Andrew
[16:43:33] <Krinkle>	 andrewbogott: cool
[16:43:35] <Krinkle>	 andrewbogott: Filed https://phabricator.wikimedia.org/T96706
[16:44:47] <wikibugs>	 6Labs, 10Continuous-Integration: Create an instance image like m1.small with 2 CPUs and 30GB space - https://phabricator.wikimedia.org/T96706#1224915 (10Andrew) This flavor should be limited to the 'integration' project, right?  Or will you need it elsewhere?
[16:52:25] <wikibugs>	 6Labs, 10Continuous-Integration: Create an instance image like m1.small with 2 CPUs and 30GB space - https://phabricator.wikimedia.org/T96706#1224943 (10Krinkle) >>! In T96706#1224915, @Andrew wrote: > This flavor should be limited to the 'integration' project, right?  Or will you need it elsewhere?  We won't...
[16:52:35] <Krinkle>	 andrewbogott: Is the migration supposed to be finished for "cvn" instances?
[16:52:47] <andrewbogott>	 Krinkle: nope, the first one is still in process.
[16:52:52] <Krinkle>	 oh, ok :)
[16:52:57] <andrewbogott>	 It must be big :)
[16:53:06] <andrewbogott>	 I’m starting one migration every 20 mins.
[16:53:20] <Krinkle>	 it's very active in terms of network/cpu.
[16:54:35] <Krinkle>	 andrewbogott: I want to minimise down time since every minute down time is a minute edits unmonitored for CVN.
[16:54:54] <Krinkle>	 But I can quite the bots and maybe you can suspend the cluster at once and move them?
[16:54:58] <Krinkle>	 quit*
[16:55:06] <Krinkle>	 Don't know how long that would take
[16:55:23] <Krinkle>	 They're running fine at the moment from what I can see so that's good.
[16:55:32] <Krinkle>	 #cvn-sw is the most active reporting channel
[16:55:53] <andrewbogott>	 Yeah, I think the actual downtime will be mininal.  It just suspends for a few seconds at the end of the copy to keep things in sync.
[16:56:04] <Krinkle>	 Ah, cool
[16:56:40] <Krinkle>	 I assume it would naturally delete any files that may have been deleted between the first copy and the last?
[16:57:42] <andrewbogott>	 yes, the move should be entirely invisible to users apart from a brief freeze
[16:58:04] <andrewbogott>	 (which is occasionally as long as a minute, but usually just a few seconds)
[16:58:10] <Krinkle>	 cool
[17:00:46] <andrewbogott>	 Krinkle: cvn-apache5 is now done moving.  Any ill effects?
[17:01:56] <Krinkle>	 https://cvn.wmflabs.org/api.php?users=MoiraMoira&pages=Template:Delete working fine
[17:01:59] <Krinkle>	 A-ok :)
[17:02:10] <andrewbogott>	 great!
[17:02:23] <andrewbogott>	 The rest should finish up within the hour — then I’ll do staging.
[17:03:12] <Krinkle>	 andrewbogott: So it moved from virt1006 to virt1007?
[17:03:23] <andrewbogott>	 to labvirt1001
[17:03:26] <Krinkle>	 wait, no that was the last one
[17:03:26] <Krinkle>	 right
[17:03:47] <Krinkle>	 wikitech would reflect this?
[17:04:18] <andrewbogott>	 No :(  I spent a while trying to force notifications for this but it turned out to be hard.
[17:04:20] <andrewbogott>	 and/or broken
[17:04:32] <bearND>	 Hello, any Special:NovaProject admin here who could add some of my colleagues as Nova_Resource:Deployment-prep members: Deskana, dr0ptp4kt, dbrant, mholloway?
[17:04:41] <andrewbogott>	 I need to hack something together that forces a labs-wide update.
[17:05:18] <Krinkle>	 andrewbogott: I'd like to delete cvn-apache7, that was an experiment to re-create apache5 as trusty. 
[17:05:27] <Krinkle>	 Which I haven't finished and will redo later
[17:05:29] <Krinkle>	 as a new one
[17:05:42] <andrewbogott>	 Krinkle: that one is mid-migration, so maybe let it be and delete in an hour?
[17:05:47] <Krinkle>	 Okay
[17:05:57] <andrewbogott>	 It would probably be harmless to delete now but I want to see this through for testing purposes.
[17:06:03] <Krinkle>	 Yeah
[17:07:07] <andrewbogott>	 bearND: I can do that, although you’ve caught us at our most distracted :)
[17:08:02] <bearND>	 andrewbogott: it's not urgent
[17:08:49] <andrewbogott>	 bearND: did I get all of them?
[17:09:32] <bearND>	 andrewbogott: Yes, thank you very much!
[17:09:41] <andrewbogott>	 sure hting
[17:10:31] <Coren>	 gifti: Thank you, I'm going to reboot it now.
[18:28:43] <SMalyshev>	 is there any way to get to the console of the labs machine? (not only for looking)
[18:31:29] <Krenair>	 SMalyshev, "the labs machine"?
[18:31:50] <SMalyshev>	 Krenair: the instance that runs in labs
[18:31:59] <SMalyshev>	 like i-00000926.eqiad.wmflabs
[18:32:18] <Krenair>	 ... I don't follow.
[18:32:47] <Krenair>	 Are you asking how to SSH to a labs instance?
[18:33:31] <legoktm>	 SMalyshev: I think wikitech exposes the console...in a weird jquery.ui popup
[18:33:46] <SMalyshev>	 legoktm: yes, but read-only
[18:34:16] <legoktm>	 andrewbogott would know then? ^
[18:35:32] <andrewbogott>	 legoktm: yeah, on the ‘manage instances’ page there’s a link ‘get console output'
[18:37:15] <legoktm>	 andrewbogott: SMalyshev is saying that it's readonly?
[18:37:22] <andrewbogott>	 yeah
[18:37:31] <andrewbogott>	 definitely no interactive console, that’s what ssh is for
[18:38:00] <SMalyshev>	 well, ssh is different (and doesn't work for me so for reason). well, I guess I just reboot the thing
[18:46:48] <andrewbogott>	 SMalyshev: sorry, all of the staff is in meetings today, so not very attentive.  Do you mean ssh doesn’t work for you in labs anywhere, or just on that instance?
[18:47:14] <SMalyshev>	 andrewbogott: no, just that one... got stuck for some reason. no worries, I rebooted it
[18:47:48] <andrewbogott>	 ok
[19:13:07] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Kanzat was created, changed by Kanzat link https://wikitech.wikimedia.org/wiki/Nova+Resource%3aTools%2fAccess+Request%2fKanzat edit summary: Created page with "{{Tools Access Request |Justification=Hi, I'd like to create a tool to help in finding templates that can be easily translated to another wikipedia language section.    For ex..."
[19:18:20] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Kanzat was modified, changed by Kanzat link https://wikitech.wikimedia.org/w/index.php?diff=155007 edit summary: 
[19:50:12] <wikibugs>	 7Tool-Labs: Use instances with resources customized for Tools use - https://phabricator.wikimedia.org/T96714#1225205 (10scfc) 3NEW
[19:56:59] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mpaa was modified, changed by Mpaa link https://wikitech.wikimedia.org/w/index.php?diff=155008 edit summary: 
[20:27:17] <wikibugs>	 6Labs, 3Labs-Q4-Sprint-2, 3Labs-Q4-Sprint-3, 3ToolLabs-Goals-Q4: Do a rolling restart of Tool Labs precise instances - https://phabricator.wikimedia.org/T95557#1225318 (10coren) 5Open>3Resolved This is done.
[20:27:19] <wikibugs>	 6Labs, 3Labs-Q4-Sprint-2, 3ToolLabs-Goals-Q4: Schedule reboot of all Labs Precise instances - https://phabricator.wikimedia.org/T95556#1225320 (10coren)
[20:41:50] <SMalyshev>	 if I use labs-vagrant, which user I should be using? 
[20:42:06] <SMalyshev>	 I'm getting PHP Warning:  dba_open(/var/cache/mediawiki/wikidata/l10n_cache-en.cdb.tmp.2146080986): failed to open stream: Permission denied in /srv/vagrant/mediawiki/vendor/wikimedia/cdb/src/Writer/DBA.php on line 38 even though it worked some time ago
[20:42:33] <bd808>	 SMalyshev: that file should be owned by www-data
[20:43:01] <SMalyshev>	 smalyshev@wdq-wikidata:/vagrant/mediawiki$ ls -ld /var/cache/mediawiki/wikidata/
[20:43:02] <SMalyshev>	 drwxrwxr-x 2 vagrant www-data 4096 Feb 27 23:19 /var/cache/mediawiki/wikidata/
[20:43:07] <SMalyshev>	 is this right?
[20:43:13] <bd808>	 looks right
[20:43:31] <SMalyshev>	 hmm... then why it worked before but not now?
[20:44:04] <SMalyshev>	 time php5 /var/www/w/MWScript.php  extensions/WikidataBuildResources/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --format ttl --limit 1000 --output f
[20:44:06] <bd808>	 is this an error from hhvm (web request) or running a script?
[20:44:13] <SMalyshev>	 script
[20:44:16] <SMalyshev>	 see above
[20:44:25] <bd808>	 ah. use /usr/local/bin/mwscript
[20:44:30] <bd808>	 it sudo's properly
[20:44:37] <bd808>	 or at least should
[20:44:46] <SMalyshev>	 sudo: a password is required
[20:44:54] <bd808>	 grr
[20:44:55] <SMalyshev>	 that's what I get from mwscript
[20:45:17] <bd808>	 that's labs puppet and mw-v's puppet fighting over a sudoers file
[20:45:53] <SMalyshev>	 it worked about a month ago but I didn't use this one for a while and now I come back and it all broken
[20:45:59] <bd808>	 in your first command, try adding a leading `sudo -u www-data -- `
[20:46:20] <bd808>	 I'll check one of my labs instances to make sure the sudoers stuff looks right
[20:46:37] <bd808>	 sometimes it gets messed up upstream in ops/puppet
[20:47:32] <SMalyshev>	 hmm... it asks me for password. Not sure which one
[20:47:50] <bd808>	 hmm. is this in beta cluster or elsewhere
[20:49:06] <bd808>	 beta cluster has weird sudoers stuff that may be set wrong for you
[20:49:17] <SMalyshev>	 bd808: I don't know. How I can find that out?
[20:49:29] <SMalyshev>	 Sorry, user smalyshev is not allowed to execute '/usr/bin/time php5 /var/www/w/MWScript.php extensions/WikidataBuildResources/extensions/Wikibase/repo/maintenance/dumpRdf.php --wiki wikidatawiki --format ttl --limit 1000 --output f' as www-data on wdq-wikidata.eqiad.wmflabs.
[20:49:53] <SMalyshev>	 that's when I give it one of the passwords which it apparently ok to accept
[20:50:19] <bd808>	 you should have passwordless sudo, but it needs to be setup in the labs project
[20:50:26] <andrewbogott>	 Krinkle: still working?
[20:50:35] <andrewbogott>	 I mean, are you there, and are you working?
[20:50:36] <bd808>	 SMalyshev: usually by adding you to the admin user group for the project
[20:50:44] <SMalyshev>	 bd808: I can do just sudo...
[20:50:56] <SMalyshev>	 smalyshev@wdq-wikidata:/vagrant/mediawiki$ sudo id
[20:50:56] <SMalyshev>	 uid=0(root) gid=0(root) groups=0(root)
[20:51:09] <bd808>	 huh. but not as www-data then?
[20:51:12] <SMalyshev>	 but I'm not sure running it under root is not going to mess stuff further
[20:51:22] <bd808>	 it will be worse
[20:51:26] <SMalyshev>	 apparently no, www-data is a special flower
[20:51:48] <bd808>	 you can do this mess `sudo -- sudo -u www-data -- ...`
[20:52:10] <bd808>	 but we should figure out the permissions problem. 
[20:52:22] <bd808>	 I can't remember how to find the project from the hostname
[20:52:31] <SMalyshev>	 I know the project
[20:52:54] <SMalyshev>	 it's wikidata-query
[20:53:13] <bd808>	 cool. looking at sudoers policies now...
[20:53:55] <SMalyshev>	 ahh... if I sudo to www-data it can't write to /vagrant/mediawiki
[20:54:56] <SMalyshev>	 I wonder if all my setup is now messed up... it all worked just fine before under my user
[20:55:31] <bd808>	 SMalyshev: Mind if I look around there a bit and see if I can figure it out?
[20:55:37] <SMalyshev>	 bd808: sure
[20:55:45] <SMalyshev>	 it's on wdq-wikidata
[20:56:01] <Krinkle>	 andrewbogott: I'm here
[20:56:07] <Krinkle>	 andrewbogott: I just rebooted some of the bots 
[20:56:10] <SMalyshev>	 aka i-000008e4.eqiad.wmflabs
[20:57:06] <bd808>	 SMalyshev: I did sudo chown www-data /vagrant/mediawiki/cache. Does that fix it for you?
[20:57:31] <SMalyshev>	 not in the original form
[20:57:37] <bd808>	 "sudo: a password is required" for mwscript still
[20:57:43] * bd808 looks deeper
[20:57:51] <SMalyshev>	 but seems to work with double sudo
[20:58:05] <andrewbogott>	 Krinkle: ok — so, I have somewhat bad news (which is probably my fault).  Two of the cvn instances (cvn-app4 and cvn-app5) use a deprecated image type which is no longer available.  So they can’t be easily transferred to a new labs host.
[20:58:19] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mpaa was modified, changed by Mpaa link https://wikitech.wikimedia.org/w/index.php?diff=155029 edit summary: Undo revision 155008 by [[Special:Contributions/Mpaa|Mpaa]] ([[User talk:Mpaa|talk]])
[20:58:29] <andrewbogott>	 Options are 1) you rebuild them 2) I shut them down and cold-migrate 3) you declare both those options to be unacceptable and I keep searching for a workaround
[20:58:29] <Krinkle>	 andrewbogott: Hm.. I thought that was all self contained?
[20:58:43] <andrewbogott>	 Krinkle: Some of it is copy-on-write.
[20:58:55] <Krinkle>	 andrewbogott: What does #2 mean
[20:59:07] <andrewbogott>	 A while ago I ran a report to show me which image types were no longe rin use and I purged them.  My report must’ve been incorrect :(
[20:59:29] <andrewbogott>	 #2 just means the instances are turned off during the copy.
[20:59:39] <Krinkle>	 that's fine, everything starts from cron
[20:59:40] <SMalyshev>	 bd808: with double sudo it kind of works... but I'm pretty sure I was able to run it under my user before
[20:59:44] <bd808>	 SMalyshev: running `labs-vagrant provision` didn't fix it either. There are definately sudoers config missing that I would expect
[20:59:53] <andrewbogott>	 Krinkle: ok — if I do them one at a time it won’t cause major project downtime?
[20:59:54] <Krinkle>	 andrewbogott: I rebooted them a minute ago in fact. For a differnet reason.
[21:00:11] <Krinkle>	 andrewbogott: Yes, you can shut down safely and then reboot on the other end.
[21:00:16] <andrewbogott>	 Krinkle: ok, that’s reassuring :)  Ok if I move them now?
[21:00:20] <Krinkle>	 Yep, go ahead.
[21:00:23] <andrewbogott>	 Will you be around for another 30 mins or so?
[21:00:41] * andrewbogott starts the move
[21:01:06] <andrewbogott>	 !log cvn cold-migrating cvn-app4 to labvirt1005
[21:01:11] <labs-morebots>	 Logged the message, dummy
[21:03:03] <bd808>	 SMalyshev: I opened https://phabricator.wikimedia.org/T96717 -- I'll look into it
[21:04:27] <SMalyshev>	 bd808: thanks
[21:17:44] <Rhamy>	 Test
[21:26:18] <Krinkle>	 !log cvn Apply role::labs::lvm::srv via Hiera
[21:26:22] <labs-morebots>	 Logged the message, Master
[21:36:13] <Coren>	 Need to reboot my box for secuirty updates.  BBIAF.
[21:40:02] <Coren>	 Back.
[21:57:57] <andrewbogott>	 Krinkle: update:  I haven’t done anything with those instances and probably won’t until tomorrow.  Ran into a side issue that I need to resolve first.
[22:03:12] <Krinkle>	 andrewbogott: OK. As long as they keep running :)
[22:03:40] <YuviPanda>	 hey Krinkle 
[22:03:52] <YuviPanda>	 Krinkle: I’m considering setting up a ‘supported’ cdnjs mirror on tools. thoughts?
[22:09:43] <wikibugs>	 6Labs, 10Continuous-Integration: Create an instance image like m1.small with 2 CPUs and 30GB space - https://phabricator.wikimedia.org/T96706#1225795 (10Andrew) 5Open>3Resolved Custom flavor ci1.medium should now be available for the Integration project.
[22:35:58] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Dgdiego was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=155053 edit summary: 
[22:39:28] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/B was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=155061 edit summary: 
[22:43:18] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Kanzat was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=155075 edit summary: 
[23:33:50] <wikibugs>	 7Tool-Labs: Provide a clone of cdnjs for toollabs users - https://phabricator.wikimedia.org/T96799#1226377 (10yuvipanda) 3NEW
[23:33:57] <YuviPanda>	 Krinkle: https://phabricator.wikimedia.org/T96799
[23:34:06] <YuviPanda>	 Krinkle: how do I provide a notification to the people behind /static?
[23:34:12] <YuviPanda>	 are you the person behind /static? :)
[23:34:30] <wikibugs>	 7Tool-Labs: Provide a clone of cdnjs for toollabs users - https://phabricator.wikimedia.org/T96799#1226387 (10yuvipanda) /static is also serving from NFS, while this will serve from local disk.
[23:35:01] <Krinkle>	 YuviPanda: I am not, but I do have partial access
[23:35:11] <Krinkle>	 YuviPanda: In what way would this be able to use local disk?
[23:35:22] <YuviPanda>	 Krinkle: because it’s going to live on the same host as tools-static and be puppetized.
[23:35:37] <YuviPanda>	 so I can turn on more optimizations (sendfile, for example)
[23:36:18] <Krinkle>	 right
[23:36:24] <Krinkle>	 tools-static wouldn't be nfs
[23:36:36] <YuviPanda>	 hmm?
[23:36:39] <YuviPanda>	 tools-static *is* NFS
[23:36:44] <YuviPanda>	 but it serves per-tool static files
[23:36:47] <YuviPanda>	 this would be tools-cdnjs
[23:36:51] <YuviPanda>	 but live on same host as tools-static
[23:37:00] <Krinkle>	 To what end?
[23:37:09] <YuviPanda>	 cdnjs?
[23:37:10] <Krinkle>	 I mean why not tools-static?
[23:37:24] <Krinkle>	 I asked before but lost track, maybe you didn't ping me in the reply, sorry
[23:37:27] <YuviPanda>	 Krinkle: basically, csteipp going ‘it would be easier to reason about crossdomain artifacts if it were a different domain'
[23:37:39] <YuviPanda>	 well, crossdomain 3rd party artifacts
[23:37:49] <YuviPanda>	 any tool user can put HTML in tools-static.wmflabs.org
[23:38:09] <Krinkle>	 Yes
[23:38:20] <Krinkle>	 but nothing executes from it server-side
[23:38:25] <Krinkle>	 so the cookies don't apply anywhere
[23:38:29] <Krinkle>	 other than *.wmflabs.org
[23:38:40] <YuviPanda>	 Krinkle: true.
[23:38:48] <Krinkle>	 which can already be done from tools.*
[23:39:20] <YuviPanda>	 Krinkle: so I don’t actually know enough about Cross Domain issues to comment. let’s move to a channel with csteipp around
[23:39:22] <Krinkle>	 we will need the exact same cross-origin waiver as for tools-static otherwise fonts don't work
[23:39:31] <Krinkle>	 This is one of those
[23:39:33] <Krinkle>	 :)
[23:39:38] <csteipp>	 Krinkle: Without knowing too much about how tools is setup, and the security requirements of the tools there, I suggested a separate domain as just a way to keep the separate in case we need to segment them for security in the future.
[23:39:41] <YuviPanda>	 oh he’s here
[23:39:47] <YuviPanda>	 :)
[23:39:56] <YuviPanda>	 Krinkle: well, it will have CORS enabled by default yeah
[23:40:22] <Krinkle>	 csteipp: tools is free sign up by anyone. tools-static is a part of that eco system where teh same users can publish static resources (meant to be cookieless)
[23:40:45] <Krinkle>	 CORS is enabled there so that people can load stylesheets and fonts from there
[23:41:14] <YuviPanda>	 tool-cdnjs would also have CORS enabled, I guess.
[23:41:26] <YuviPanda>	 so the alternative to tools-cdnjs.wmflabs.org would be tools-static.wmflabs.org/cdnjs/*
[23:41:35] <Krinkle>	 It's a bit like wikimediausercontent.org in that we only need one of them, not many.
[23:41:39] <csteipp>	 Krinkle: So people don't just include a src= to there, but load them in via xhr?
[23:41:45] <YuviPanda>	 they do src= too :)
[23:41:52] <Krinkle>	 csteipp: Yeah, they load it as anything else.
[23:42:11] <Krinkle>	 But the scripts hosted there (e.g. bootstrap, jquery ui) request additional resources.
[23:42:18] <Krinkle>	 and browsers require CORS for those to work
[23:43:08] <Krinkle>	 a stylesheet resolves resources relative against itself, not relative against the main browser window. This is a feauture so that one can publish a stylesheet with images/fonts/other stylesheets next to it without it needing to know its external address.
[23:43:34] <YuviPanda>	 Krinkle: an advantage of tools-cdnjs would be an index page of sorts on the frontpage, I guess.
[23:43:48] <Krinkle>	 anyway, tools-static is essentially meant as a CDN. and 'cdnjs' would be just another user of that infrastructure.
[23:43:54] <csteipp>	 Krinkle: Right... so stuff on cdnjs is loading in stuff that you would want to host on tools-static?
[23:44:01] <Krinkle>	 YuviPanda: that should be on tools.wmflabs.org/cdnjs
[23:44:12] <Krinkle>	 YuviPanda: We can redirect / to tools.
[23:44:23] <Krinkle>	 of subdir
[23:44:27] <YuviPanda>	 yeah, but that’s going to be somewhat difficult to do (cdnjs repo is 11G :P)
[23:44:31] <YuviPanda>	 but we can figure a way out
[23:44:32] <YuviPanda>	 sure
[23:44:49] <Krinkle>	 YuviPanda: Wait, why would the index not work. You said html works?
[23:44:58] <YuviPanda>	 Krinkle: the cdnjs isn’t html, no?
[23:45:15] <YuviPanda>	 it’s this npm app that calls out to disqus, and some other thirdparty service for indexing...
[23:45:34] <Krinkle>	 Yes, but it's not on the same domain at cloudfare either
[23:45:49] <Krinkle>	 cdnjs.cloudflare.com vs cdnjs.com
[23:45:55] <YuviPanda>	 hmm, fair enough
[23:46:07] <Krinkle>	 which makes sense since it's supposed to be static
[23:46:20] <Krinkle>	 csteipp: cdnjs doesn't interact explicitly with tools-static. 
[23:46:49] <Krinkle>	 csteipp: But cdnjs is essentially just a bundle of static resources, much like the bundles users already serve from tools-static.
[23:46:54] <Krinkle>	 It wouldn't be any different. 
[23:46:58] <YuviPanda>	 I don’t feel strongly about it, so if you can convince csteipp that it’s ok to have it in tools-static.wmflabs.org/cdnjs, then I’m totally cool with just putting it there
[23:48:07] <YuviPanda>	 Krinkle: ^
[23:48:27] <csteipp>	 Krinkle: My recommendation was based on about 3 minutes of Yuvi explaining cdnjs to me. If you've thought this through, don't let me block you. I'm still not clear why having one domain would be better, or what the CORS relationships would need to be.
[23:49:18] <Krinkle>	 csteipp: I haven't thought this through, but I'm not sure what the benefit is of setting up a separate domain.
[23:49:43] <Krinkle>	 What difference do you believe there is, or what separation are we trying to signify?
[23:50:13] <YuviPanda>	 The separation between ‘any tool can put HTML / CSS / JS here’ vs ‘This is just straight up what is from cdnjs'
[23:50:50] <csteipp>	 Krinkle: So if a script on cdnjs introduces a dom-xss, I was thinking having a separate domain lets us keep the service up without risking anything that might be on tools-static... under the assumption that there might be something private on tools-static, but no one really knows everything that runs there.
[23:51:33] <Krinkle>	 csteipp: tools-static is purely static. that is enforced by the web server. No server-side executables.
[23:52:18] <Krinkle>	 Note that tools are not able to publish their resources just "anywhere" on tools-static, each tool has a restricted sub directory.
[23:52:41] <csteipp>	 Krinkle: Hmm, yeah, and I guess a dom xss would be on the including domain anyway... so yeah, not much separation there.
[23:52:53] <Krinkle>	 Imagine someone other than us creating a "cdnjs" tool account now, and publishing it there (nothing is us from stopping them from doing that for us).
[23:53:28] <YuviPanda>	 (I already created cdnjs) :D
[23:53:31] <Krinkle>	 and users will no doubt have their own bundles even if we provide cdnjs.
[23:53:42] <csteipp>	 Krinkle: Is there ever a reason someone would want to include only cdnjs from us, but not all of tools-static?
[23:53:46] <Krinkle>	 So then the question becomes, do we benefit from special-casing it
[23:54:14] <csteipp>	 If they're separate domains, then you can enforce that with CSP
[23:54:47] <Coren>	 Anything on tools-static is - by definition - just static files.  I'm not sure I see a possible attack vector.
[23:55:23] <Krinkle>	 csteipp: the domain has no executable web apps. the shared origin *.wmflabs.org is already accessible by tools.wmflabs.org by the same authors.
[23:56:09] <Krinkle>	 Tool authors sometimes have all their resources inside their tools.wmflabs.org subdir, some also mount some at tools-static, some would make use of our first-party cdnjs account
[23:56:21] <Krinkle>	 but it's all third-party anyway.
[23:56:40] <Krinkle>	 cdnjs is not curated in that way like other cdnjs. Essentially straight out of npm.
[23:56:53] <Krinkle>	 other cdns*
[23:57:22] <YuviPanda>	 Krinkle: it’s somewhat curated, I think. not particularly well, of course :)
[23:57:23] <YuviPanda>	 but still
[23:57:28] <YuviPanda>	 it’s not a straight up npm mirror
[23:58:05] <Krinkle>	 YuviPanda: I maintain jquery-json package. I push to github repo now and to npm, it'll be live on cdnjs cloudfare in minutes with no review.
[23:58:16] <YuviPanda>	 Krinkle: true but initially at least 
[23:58:19] <YuviPanda>	 there’s ‘curation'
[23:58:25] <YuviPanda>	 it’s not curated like how debian packages are
[23:58:36] <YuviPanda>	 but at least you’re not going to get random non JS / CSS things..
[23:59:04] <Krinkle>	 sure, that's true