[00:01:22] <matt_flaschen>	 quiddity, wherever you got that from (I might have written it), it's now ssh login.tools.wmflabs.org:
[00:01:54] <quiddity>	 matt_flaschen, from https://en.wikipedia.org/wiki/MediaWiki_talk:Gadget-BugStatusUpdate.js#Not_working
[00:05:26] <matt_flaschen>	 quiddity, restarted and confirmed gadget is working again.
[00:12:11] <quiddity>	 ty!
[02:19:01] <yuvipanda>	 matt_flaschen want to move it to kubernetes? should make it more stable.
[02:31:33] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Build replacement for the webservice toolschecker test - https://phabricator.wikimedia.org/T142164#2525647 (10yuvipanda)
[03:07:00] <wikibugs>	 06Labs, 10Labs-Infrastructure: Super weird networking behavior - https://phabricator.wikimedia.org/T142165#2525709 (10yuvipanda)
[03:09:10] <wikibugs>	 06Labs, 10Labs-Infrastructure: Super weird networking behavior - https://phabricator.wikimedia.org/T142165#2525721 (10yuvipanda) This behavior (wait forever then timeout) is consistent with what happens when there are security groups issues present, except there are no security between these nodes - they are a...
[03:21:09] <wikibugs>	 06Labs, 10Labs-Infrastructure: Create a new labs flavor available to all project: largedisk - https://phabricator.wikimedia.org/T142166#2525722 (10yuvipanda)
[03:21:48] <wikibugs>	 06Labs, 10Tool-Labs: Change ordering of image flavors in wikitech for tools - https://phabricator.wikimedia.org/T142167#2525735 (10yuvipanda)
[03:36:55] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Build replacement for the webservice toolschecker test - https://phabricator.wikimedia.org/T142164#2525748 (10yuvipanda) For kubernetes, we should do the following  1. Spawn a simple pod with a built in simple web server, no NFS, hit its IP to see...
[03:58:25] <wikibugs>	 06Labs, 10Labs-Infrastructure: Super weird networking behavior - https://phabricator.wikimedia.org/T142165#2525749 (10yuvipanda) ``` root@tools-puppetmaster-01:/home/yuvipanda# sudo tcpdump src 10.68.21.13 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type E...
[04:06:53] <wikibugs>	 06Labs, 10Labs-Infrastructure: Super weird networking behavior - https://phabricator.wikimedia.org/T142165#2525753 (10yuvipanda) I restarted nova-network, no change in behavior.
[04:37:20] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/ProgrammingGeek was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=816404 edit summary: 
[04:37:23] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Heikki-hakala was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=816405 edit summary: 
[04:37:26] <wm-bot>	 Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Bowleerin was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=816406 edit summary: 
[05:51:55] <shinken-wm>	 PROBLEM - Puppet staleness on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[07:03:39] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: graphite.wmflabs.org is very slow / flaky - https://phabricator.wikimedia.org/T127957#2525925 (10MoritzMuehlenhoff)
[07:03:41] <wikibugs>	 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Copy labmon data to new SSDs - https://phabricator.wikimedia.org/T137924#2525923 (10MoritzMuehlenhoff) 05Resolved>03Open The temp host, which was added in https://phabricator.wikimedia.org/rOPUP93cef36f9ed6e2ec1e8ec2f6d5d345a4343e1610 is still up and run...
[07:40:41] <he1kki>	 Does tools-login.wmflabs.org ssh login use authorized_keys under ~/.ssh?
[07:42:49] <he1kki>	 My initial ssh key seems to be inserted into ldap, and I don't know how to add another one into ldap.
[08:16:42] <tom29739>	 he1kki: you use the OpenStack tab in your wiki tech preferences
[08:27:20] <he1kki>	 tom29739: thanks.
[08:29:00] <he1kki>	 ed25519 public keys are "unknown format", but they are a bit unusual anyway
[08:30:08] <tom29739>	 he1kki: it only accepts certain types I think
[08:30:23] <tom29739>	 The "normal" ones it accepts
[08:30:57] * tom29739 doesn't know much about SSH keys
[08:31:15] <he1kki>	 yeap, id_rsa was accepted without issues
[10:41:59] <wikibugs>	 06Labs: (Re-)Create Gitblit->Phabricator testing instance on Labs - https://phabricator.wikimedia.org/T142186#2526306 (10Danny_B)
[15:27:37] <wikibugs>	 06Labs: Enable root passwords on Labs VMs - https://phabricator.wikimedia.org/T142216#2527541 (10Danny_B)
[15:54:46] <wikibugs>	 06Labs: Designate seems broken in labtest - https://phabricator.wikimedia.org/T142220#2527607 (10AlexMonk-WMF)
[15:55:10] <wikibugs>	 06Labs: Designate seems broken in labtest - https://phabricator.wikimedia.org/T142220#2527620 (10AlexMonk-WMF)
[16:01:09] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 032 V: 032] Set ALLOWED_HOSTS to ['*'] by default [labs/striker] - 10https://gerrit.wikimedia.org/r/301904 (owner: 10BryanDavis)
[16:07:58] <grrrit-wm>	 (03PS1) 10BryanDavis: Bump striker to 5c1e9cf [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/303182 
[16:08:15] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 032 V: 032] Bump striker to 5c1e9cf [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/303182 (owner: 10BryanDavis)
[16:11:35] <halfak>	 !log ores terminated ores-web-04 (re-creating instance with medium image size)
[16:11:38] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master
[16:15:38] <halfak>	 !log ores creates ores-web-04 debian-8.5-jessie
[16:15:41] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master
[16:29:52] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Enable access to Wikipedia Tulu (tcywiki) on labs replicas - https://phabricator.wikimedia.org/T142223#2527737 (10jcrespo)
[16:30:24] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Enable access to Wikipedia Tulu (tcywiki) on labs replicas - https://phabricator.wikimedia.org/T142223#2527756 (10jcrespo) a:03jcrespo I have first to apply the filters on production side.
[16:48:03] <yuvipanda>	 ottomata hey! did the network weirdness you were seeing yesterday go away? https://phabricator.wikimedia.org/T142165 I'm still seeing that
[16:51:54] <ottomata>	 yuvipanda:  ah sorry got caught up didn't create a ticket
[16:51:57] <ottomata>	 it did eventually go away
[16:52:12] * yuvipanda nods
[16:52:21] <yuvipanda>	 ottomata does my description sound similar?
[16:55:20] <ottomata>	 ja will comment.
[16:56:06] <yuvipanda>	 ottomata cool
[16:57:30] <wikibugs>	 06Labs, 10Labs-Infrastructure: Super weird networking behavior - https://phabricator.wikimedia.org/T142165#2527830 (10Ottomata) I encountered simliar behavior yesterday in deployment-prep.  I created deployment-kafka05.  It needed to talk to deployment-zookeeper01.  For the first (at least) 15 minutes of its l...
[17:01:34] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Enable access to Wikipedia Tulu (tcywiki) on labs replicas - https://phabricator.wikimedia.org/T142223#2527858 (10jcrespo) a:05jcrespo>03None ``` $ for host in labsdb1001.eqiad.wmnet labsdb1003.eqiad.wmnet; do mysql -h $host -e "SELECT count(*) FROM t...
[17:03:29] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Enable access to Wikipedia Tulu (tcywiki) on labs replicas - https://phabricator.wikimedia.org/T142223#2527737 (10AlexMonk-WMF) ```krenair@tools-bastion-03:~$ sql tcywiki ERROR 1049 (42000): Unknown database 'tcywiki_p'``` Needs a maintain-replicas run
[17:15:38] <yuvipanda>	 valhallasw: legoktm do you know when the toolserver started?
[17:19:24] <valhallasw`vecto>	 Yuvipanda: uh
[17:20:54] <valhallasw`vecto>	 https://lists.wikimedia.org/pipermail/toolserver-l/2005-October/thread.html
[17:21:16] <yuvipanda>	 so at least 2005
[17:22:08] <valhallasw`vecto>	 https://web.archive.org/web/20051124135226/http://tools.wikimedia.de/
[17:22:17] <valhallasw`vecto>	 I think 05-ish
[17:22:38] <valhallasw`vecto>	 There's no older aechive.org copy
[17:23:12] <yuvipanda>	 right
[17:23:15] <yuvipanda>	 ok
[17:24:34] <valhallasw`vecto>	 Why do you want to know?
[17:25:13] <yuvipanda>	 valhallasw: the kubernetes people wanted an estimate of how long this community has been a thing of sorts, so ~10y seems appropriate number
[17:25:22] <valhallasw`vecto>	 Right
[17:25:44] <yuvipanda>	 valhallasw btw, we now have log aggregation for all tools nodes in tools-logs-01 :) look at /srv/syslog/syslog.log :D
[17:26:07] <valhallasw`vecto>	 Ooooh
[17:26:08] <Betacommand>	 yuvipanda: its been at least 04
[17:26:23] <valhallasw`vecto>	 Syslog, or also puppet
[17:26:53] <yuvipanda>	 valhallasw: it's collecting and keeping for only3d just now, but I'll  increase that over time.
[17:27:01] <yuvipanda>	 valhallasw: puppet also writes into syslog I think
[17:27:09] <valhallasw`vecto>	 Ah, cool
[17:27:54] <yuvipanda>	 valhallasw: am verifying now
[17:28:04] <yuvipanda>	 valhallasw: yup, if you tail and grep for puppet-agent
[17:28:11] <yuvipanda>	 it shows up
[17:28:33] <valhallasw`vecto>	 Cool
[17:38:53] <wikibugs>	 10Tool-Labs-tools-phabricator-bug-status: Move phabricator-bug-status to kubernetes - https://phabricator.wikimedia.org/T142237#2528144 (10Mattflaschen-WMF)
[17:39:15] <wikibugs>	 10Tool-Labs-tools-phabricator-bug-status: Move phabricator-bug-status to kubernetes - https://phabricator.wikimedia.org/T142237#2528161 (10Mattflaschen-WMF)
[17:39:47] <matt_flaschen>	 yuvipanda, thanks for the suggestion.  Filed T142237 .
[17:39:47] <stashbot>	 T142237: Move phabricator-bug-status to kubernetes - https://phabricator.wikimedia.org/T142237
[17:44:09] <wikibugs>	 06Labs, 10Labs-Infrastructure: Super weird networking behavior - https://phabricator.wikimedia.org/T142165#2528199 (10yuvipanda) I can continue to hit another node that has different security groups applied but on the same labvirt host (tools-worker-1008 and tools-docker-builder-01), but not other node on diff...
[17:44:29] <halfak>	 !log ores repooled ores-web-04 in hiera and ran puppet on ores-lb-02
[17:44:32] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master
[18:07:34] <bd808>	 Is the Tools http proxy dead/dying? How do I debug that?
[18:08:15] <bd808>	 yuvipanda: ^ really really long wait and then a 504 on different tools.wmflabs.org urls
[18:08:27] <yuvipanda>	 hmm, look at tools-proxy-01 and see
[18:08:37] <yuvipanda>	 looking now
[18:08:47] <yuvipanda>	 but that shouldn't produce 504
[18:09:16] <shinken-wm>	 PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:09:17] <bd808>	 https://tools.wmflabs.org/bd808-test/, https://tools.wmflabs.org/sal/, https://tools.wmflabs.org/plagiabot/, ...
[18:09:38] <yuvipanda>	 yeah, I see it
[18:09:41] <yuvipanda>	 'connection refused'
[18:10:07] <yuvipanda>	 andrewbogott chasemp ^ this is happening to other instances that are pre-existing now
[18:10:21] <yuvipanda>	 andrewbogott and causing tools failures :|
[18:10:31] <andrewbogott>	 yuvipanda: that's because… why are we talking in a different room now
[18:10:31] <andrewbogott>	 rather than the one that I just said that in?
[18:10:44] <Krenair>	 Luke just mentioned connection refused in deployment-prep in -releng
[18:10:58] <yuvipanda>	 oh, because bd808 pinged me here.
[18:11:20] <Krenair>	 ah
[18:11:21] <yuvipanda>	 <andrewbogott> I am causing a brief outage, working on it
[18:11:21] <andrewbogott>	 deployment-prep definitely doesn't have anything to do with what I'm doing
[18:11:24] <yuvipanda>	 bd808 Krenair ^
[18:11:25] <Luke081515>	 here is the error of beta-scap: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/114244/console
[18:11:29] <chasemp>	 inbound to all of tools is wonky atm then
[18:11:36] <yuvipanda>	 ok
[18:11:47] <chasemp>	 tools checker is going to start paging soon too 
[18:12:11] <Krenair>	 icinga already noticed a problem
[18:12:11] <yuvipanda>	 I'll mark it as downtimed
[18:12:13] <chasemp>	 yeah it's hosed
[18:13:09] <yuvipanda>	 I downtimed it for 2h
[18:13:49] <yuvipanda>	 communication inside tools is also dying, am getting cron errors now because gridengine clients can't talk to master.
[18:15:58] <Luke081515>	 hm
[18:16:10] <Luke081515>	 tools-exec-1406 has a floating IP?
[18:18:01] <Krenair>	 yep
[18:18:44] <liangent>	 tool lab cron is spamming my mailbox again?
[18:18:50] <Luke081515>	 just htis one, or other tool-exec instances too?
[18:18:57] <yuvipanda>	 yes tools is having network trouble
[18:19:00] <yuvipanda>	 all of tools
[18:19:01] <yuvipanda>	 hang on tight
[18:19:11] <liangent>	 error: unable to send message to qmaster using port 6444 on host "tools-grid-master.tools.eqiad.wmflabs": got send time our
[18:19:21] * bd808 respects the fasten seatbelt sign
[18:20:06] <yuvipanda>	 can someone not me set the topic? Matrix doesn't have topic sync yet :|
[18:20:25] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1007 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[18:20:30] <chasemp>	 yuvipanda: I don't have perms anymore
[18:22:12] <Luke081515>	 what about assining the +t flag to *!*@wikpedia/* etc? Than they can use chanserv commands to change it
[18:22:31] <yuvipanda>	 should get rid of +t I think, but no idea who added +t?
[18:22:42] <paladox>	 They added it due to spam
[18:22:48] <Luke081515>	 topic changeing issues
[18:22:54] <paladox>	 Someone spammed this channel and change the topic to some random thing
[18:23:04] <paladox>	 change = changed
[18:25:10] <yuvipanda>	 well, let's just have more people around than lock things down :) can we get rid of it?
[18:25:19] <Krenair>	 andrew can fix it
[18:25:31] <Krenair>	 think he's a bit busy though
[18:26:06] <paladox>	 Yes
[18:26:13] <paladox>	 We can get rid of +t
[18:27:00] <Steinsplitter>	 makes sense
[18:27:02] <yuvipanda>	 but yeah, later
[18:27:04] <yuvipanda>	 (I emailed labs-l)
[18:27:10] <paladox>	 ok
[18:27:20] <paladox>	 thanks
[18:34:04] <shinken-wm>	 RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.027 second response time
[18:35:14] <bd808>	 tools.wmflabs.org looks like it is working again :)
[18:35:50] <yuvipanda>	 yeah, error rate just dropped as well
[18:36:37] <doctaxon>	 cool
[18:36:40] <doctaxon>	 thank you
[18:45:26] <shinken-wm>	 RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:46:27] <tom29739>	 yuvipanda: you can remove -t, you're an op
[19:04:06] <shinken-wm>	 PROBLEM - Puppet run on tools-worker-1025 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[19:12:21] <grrrit-wm>	 (03PS1) 10Mattflaschen: Add Edit Review Improvements to #collaboration-team [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/303251 
[19:25:15] <paladox>	 yuvipanda hi, im wondering can i add tom29739 to lolrrrit-wm please?
[19:25:22] <paladox>	 he is helping me debug it
[19:25:36] <paladox>	 since it seems it keeps disconnecting every few hours
[19:26:27] <yuvipanda>	 feel free
[19:26:34] <paladox>	 Thanks
[19:28:59] <paladox>	 !log tools adding tom29739 to lolrrit-wm project
[19:29:04] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master
[19:29:29] <tom29739>	 paladox: wrong log again
[19:29:41] <paladox>	 Oh
[19:29:58] <tom29739>	 You do it next to every time :D
[19:29:59] <paladox>	 !log tools.lolrrit-wm adding tom29739 to lolrrit-wm project
[19:30:02] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master
[19:30:07] <paladox>	 Oh i only did it this time
[19:30:27] <paladox>	 since i thought we had to log in tools project when adding users to a project in tools
[19:30:31] <paladox>	 but seems i was wrong
[19:30:32] <paladox>	 sorry
[19:32:31] <tom29739>	 paladox: there's no harm in it
[19:32:40] <paladox>	 Ok
[19:36:48] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2528635 (10chasemp)
[19:41:00] <paladox>	 yuvipanda hi, is there a way we can get npm2* on lolrrit-wm please?
[19:41:06] <paladox>	 since it is running 1.x
[19:41:28] <paladox>	 and i belive there is problems with npm 1.x, so maybe 2.x will fix things
[19:41:30] <paladox>	 :)
[19:42:00] <tom29739>	 paladox: run it in k8s? If that doesn't have 2.x, then you're stuffed, it's not available
[19:42:12] <paladox>	 Oh how do you do that
[19:42:20] <yuvipanda>	 file a bug I'll look at it next week. and yes, you need to check / do all things inside 'webservice --backend=kubernetes nodejs shell'
[19:42:20] <paladox>	 does k8s have an app sotre
[19:42:22] <yuvipanda>	 and not directly on bastion
[19:42:28] <paladox>	 Oh thanks
[19:42:40] <paladox>	 I guess i will do npm upgrade and see what it will do
[19:42:52] <tom29739>	 paladox: think it's already in k8s
[19:43:03] <tom29739>	 paladox: I'm looking at it
[19:43:05] <paladox>	 Oh
[19:43:40] <paladox>	 tom29739 how do we do that
[19:43:49] <paladox>	 since npm-debug.log shows npm@1*
[19:43:55] <paladox>	 please
[19:43:56] <paladox>	 ?
[19:44:18] <yuvipanda>	 (going afk now, will be back on Monday :D)
[19:44:23] <paladox>	 Ok
[19:44:27] * tom29739 waves
[19:45:56] <paladox>	 tom29739 when i do npm -v it shows 1.4*
[19:46:33] <paladox>	 I guess we could do npm install npm@2
[19:47:10] <tom29739>	 paladox: anything that requires npm 2
[19:47:16] <tom29739>	 Just don't go there
[19:47:22] <tom29739>	 I've tried
[19:47:27] <paladox>	 Nope, but npm 1 we found kept breaking in ci
[19:47:30] <paladox>	 with node 4
[19:51:08] <tom29739>	 paladox: I'll just look at it now
[19:51:14] <paladox>	 Ok
[19:51:17] <tom29739>	 If my phone wants to work
[19:51:20] <paladox>	 Ok
[19:51:23] <paladox>	 thanks
[19:54:22] <wikibugs>	 06Labs, 10grrrit-wm: Update npm to 2.x on lolrrit-wm - https://phabricator.wikimedia.org/T142253#2528715 (10Paladox)
[19:54:29] <paladox>	 I filled https://phabricator.wikimedia.org/T142253
[20:00:14] <wikibugs>	 06Labs, 10Tool-Labs, 10grrrit-wm: Update npm to 2.x on tools - https://phabricator.wikimedia.org/T142253#2528733 (10tom29739)
[20:00:56] <wikibugs>	 06Labs, 10Tool-Labs, 10grrrit-wm: Update npm to 2.x on tools - https://phabricator.wikimedia.org/T142253#2528737 (10yuvipanda) Are you sure the version of npm *inside* kubernetes is too old?
[20:01:38] <wikibugs>	 06Labs, 10Tool-Labs, 10grrrit-wm: Update npm to 2.x on tools - https://phabricator.wikimedia.org/T142253#2528742 (10Paladox) doing webservice --backend=kubernetes nodejs  and then npm -v shows it is using 1.4
[20:02:19] <wikibugs>	 06Labs, 10Tool-Labs, 10grrrit-wm: Update npm to 2.x on tools - https://phabricator.wikimedia.org/T142253#2528743 (10yuvipanda) You need to do  ` webservice --backend=kubernetes nodejs shell`
[20:03:06] <wikibugs>	 06Labs, 10Tool-Labs, 10grrrit-wm: Update npm to 2.x on tools - https://phabricator.wikimedia.org/T142253#2528744 (10Paladox) Yep, I did that, and it shows npm 1.4
[20:04:02] <wikibugs>	 06Labs, 10Tool-Labs, 10grrrit-wm: Update npm to 2.x on tools - https://phabricator.wikimedia.org/T142253#2528746 (10Paladox) tools.lolrrit-wm@interactive:~$ npm -nv o1.4.21
[20:31:24] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2528812 (10chasemp) p:05Triage>03High
[20:32:13] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2525709 (10chasemp) a:03Andrew I'm helping on this as well but a lot of it falls into the realm of openstack internals @andrew is best suited to sort out.
[20:41:56] <wikibugs>	 10Labs-Vagrant, 10Fundraising-Backlog, 10MediaWiki-Vagrant, 07Easy: Make it easier to use the fundraising puppet role on labs - https://phabricator.wikimedia.org/T102304#2528893 (10DStrine)
[20:43:25] <wikibugs>	 06Labs, 10Labs-Infrastructure: Default source group allowances do not work post Liberty upgrade - https://phabricator.wikimedia.org/T142165#2528946 (10yuvipanda) I've disabled:  1. All access to Special:NovaSecurityGroup 2. Add instance button on Special:NovaInstance  on wikitech with a Mediawiki:Common.js hac...
[20:47:38] <wikibugs>	 10Labs-Vagrant, 10Fundraising-Backlog, 10MediaWiki-Vagrant, 07Easy: Make it easier to use the fundraising puppet role on labs - https://phabricator.wikimedia.org/T102304#1361955 (10Dzahn) Text says it was "tough", then project gets added called "easy".  Making it easier to use is probably not that easy....
[20:49:34] <wikibugs>	 10Labs-Vagrant, 10Fundraising Tech Backlog, 10Fundraising-Backlog, 10MediaWiki-Vagrant: Write MW-Vagrant puppet to allow us to spin up dev, staging, and testing instances, and deploy sandbox servers on WMF-labs - https://phabricator.wikimedia.org/T99957#2529016 (10DStrine)
[21:06:10] <tom29739>	 yuvipanda: you fixed the way to get around instance creation
[21:06:23] <tom29739>	 *instance creation being disabled
[21:06:47] <paladox>	 tom29739 it is easy to get around it
[21:06:57] <tom29739>	 paladox: how?
[21:07:00] <paladox>	 since he did it through the mediawiki:common.js page
[21:07:32] <tom29739>	 Before I couldn't create instances through horizon, but I could through wiki tech
[21:07:45] <tom29739>	 (with instance creation being disabled)
[21:07:49] <paladox>	 Yeh he did it with js
[21:07:52] <mutante>	 instance creation is disabled ..again?
[21:08:02] <paladox>	 Yep
[21:08:06] <paladox>	 an email has been sent out
[21:08:07] <mutante>	 ugh
[21:08:22] <paladox>	 for https://phabricator.wikimedia.org/T142165
[21:08:24] <paladox>	 mutante ^^
[21:08:52] <mutante>	 well then. i was about to re-create the git.wmflabs that we should not have deleted
[21:08:57] <mutante>	 as requested by Danny
[21:08:59] <paladox>	 Oh
[21:09:04] <mutante>	 then not..
[21:09:09] <paladox>	 mutante you can request it by admins
[21:10:09] <tom29739>	 mutante: disable JavaScript and use the "add instance" link on https://wikitech.wikimedia.org/wiki/Special:NovaInstance
[21:11:14] <tom29739>	 Or alternatively use the special link: https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=create&project=privpol-captcha&region=eqiad
[21:11:35] <tom29739>	 (replace the project with the one you want to create an instance for)
[21:11:48] <paladox>	 lol
[21:12:03] <tom29739>	 But I think they disabled it because there's a problem security groups on new instances
[21:12:09] <paladox>	 yep
[21:12:29] <chasemp>	 tom29739: seriously don't do that we disabled things for everyones benefit
[21:12:39] <chasemp>	 trying to get around it just means we have to be jerks for no reason
[21:13:11] <tom29739>	 I generally respect the instance creation being disabled
[21:13:50] <tom29739>	 If there wasn't a reason for it being disabled then I figure you wouldn't disable it
[21:14:47] <chasemp>	 I don't know that putting instructions to side step it in the channel is respecting it :)
[21:14:50] <chasemp>	 but thank you
[21:21:15] <paladox>	 chasemp, we should really add support for this in the extension
[21:21:33] <paladox>	 so when we flip the disabled switch in horizon then it should do the same for wikitech
[21:25:19] <chasemp>	 we hope to do this almost never but sure
[21:26:02] <paladox>	 Oh
[21:26:23] <paladox>	 chasemp is there a way we can just support a config, that shuts of instance creation in wikitech
[21:26:30] <paladox>	 or would that break every thing?
[21:26:45] <chasemp>	 no time to talk paladox :) can file an issue if you like
[21:26:55] <paladox>	 Oh ok
[21:26:56] <paladox>	 sorry
[21:39:52] <tom29739>	 Channel really needs to be -t
[21:40:10] <tom29739>	 Then we could put that instance creation is disabled in the topic
[21:40:12] <Matthew_>	 tom29739: Uh... no.
[21:40:25] <Matthew_>	 It's +t because a certain fond troll has a habit of changing topics.  
[21:40:29] <Luke081515>	 #wikimedia-labs(+n)
[21:40:32] <Luke081515>	 no t here
[21:40:33] <paladox>	 tom29739 that was done
[21:40:38] <paladox>	 theres no t
[21:40:48] <Luke081515>	 just +n
[21:40:54] <Matthew_>	 Oops, sorry.  I was confused with -tech
[21:40:59] <tom29739>	 It was..
[21:41:07] <paladox>	 yep
[21:41:17] <tom29739>	 I'm too tired
[21:41:23] <paladox>	 oh
[21:41:33] <paladox>	 but it is only 11:41 pm there
[21:41:34] <paladox>	 lol
[21:41:49] <shinken-wm>	 PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[21:46:08] <Krenair>	 It was +t earlier
[21:46:10] <Krenair>	 we removed it
[21:58:52] <paladox>	 chasemp sorry for noise, but why did we upgrade to liberty
[21:59:01] <paladox>	 since there is Mitaka
[21:59:10] <paladox>	 and liberty is about to become end of life
[22:00:54] <mutante>	 paladox: you could comment on https://phabricator.wikimedia.org/phame/post/view/10/labs_openstack_upgrade_on_tuesday_2016-08-02_16_00_utc/
[22:01:04] <mutante>	 noticed the blog post has comment feature
[22:01:05] <paladox>	 Oh ok
[22:01:07] <paladox>	 i will
[22:01:08] <paladox>	 thanks
[22:01:26] <paladox>	 Oh i carnt write a comment
[22:02:37] <mutante>	 oh, you dont even get the form?
[22:02:38] <mutante>	 ticket about the upgrade?
[22:02:53] <paladox>	 Nope
[22:04:33] <paladox>	 It seems they have disabled the comment feature
[22:05:43] <mutante>	 uhm..
[22:05:44] <mutante>	 the comment form should probably be active for all
[22:05:44] <mutante>	 i dont know about the permission system 
[22:07:10] <paladox>	 oh
[22:09:37] <Krenair>	 I can comment there
[22:09:47] <mutante>	 :/ tried
[22:09:48] <mutante>	 seems like we should have this https://phabricator.wikimedia.org/T104586
[22:09:49] <mutante>	 but for liberty
[22:10:19] <paladox>	 mutante we are already are on liberty
[22:10:38] <paladox>	 oh you mean
[22:10:41] <paladox>	 a tracking task
[22:10:42] <paladox>	 yeh
[22:10:53] <paladox>	 Krenair how can you comment
[22:10:58] <paladox>	 your an admin
[22:11:04] <paladox>	 that's probaly why
[22:11:17] <Krenair>	 I'm not really a phabricator admin
[22:11:33] <Krenair>	 I have some misc permissions but not proper admin
[22:12:10] <paladox>	 Oh
[22:13:29] <mutante>	 yea, i am surprised there is none
[22:13:38] <mutante>	 because i wanted to tell you to go there instead
[22:13:47] <mutante>	 if the blog post doesnt allow comments that is
[22:14:00] <mutante>	 i am also not a phabricator admin
[22:14:10] <mutante>	 and i have the comment field
[22:15:01] <mutante>	 a major upgrade that like that probably would get more comments
[22:15:35] <paladox>	 Oh
[22:16:52] <wikibugs>	 10Labs-project-Librarybase, 10Reports-bot, 10The-Wikipedia-Library, 10WikiCite: Create recommendations for databases/journals/websites, by WikiProject for WikiProject X - https://phabricator.wikimedia.org/T111066#2529507 (10Harej) I have submitted a grant proposal relevant to this task: https://meta.wikime...
[22:21:51] <shinken-wm>	 RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:22:39] <mutante>	 paladox: https://lists.wikimedia.org/pipermail/labs-announce/2016-August/000137.html
[22:22:46] <paladox>	 Thanks
[22:52:09] <paladox>	 !log git live testing gerrit-test3 changes things and disabling puppet for temp now
[22:52:13] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL, Master
[23:05:02] <wikibugs>	 06Labs: (Re-)Create Gitblit->Phabricator testing instance on Labs - https://phabricator.wikimedia.org/T142186#2529572 (10Dzahn) Hey @Danny_B,  sooo a couple things with this. Back when we created that we still had gitblit on antimony and gerrit on ytterbium. And the task was to shut those down.  Meanwhile they a...
[23:53:48] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/Owner-only consumers was modified, changed by Tgr (WMF) link https://www.mediawiki.org/w/index.php?diff=2209036 edit summary: /* See also */ Manual:Pywikibot/OAuth