[00:04:40] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: decom old wikitech-static machine - https://phabricator.wikimedia.org/T129391#2152577 (10Dzahn) I deleted the VM in rackspace web UI, it's gone.
[00:06:03] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: decom old wikitech-static machine - https://phabricator.wikimedia.org/T129391#2152580 (10Dzahn)
[00:06:20] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: Update wikitech-static OS/PHP version - https://phabricator.wikimedia.org/T126385#2152582 (10Dzahn)
[00:06:25] <wikibugs>	 6Labs, 6Operations, 10wikitech.wikimedia.org: decom old wikitech-static machine - https://phabricator.wikimedia.org/T129391#2104237 (10Dzahn) 5Open>3Resolved
[00:13:22] <Luke081515>	 Can somone help me? I want to set an alias for arc, phabricator.com says I have to run export PATH="$PATH:/srv/arcanist/bin/"
[00:13:35] <Luke081515>	 but when I'm logging in again, the setting is gone
[00:17:39] <Cyberpower678>	 kaldari, :-)
[00:51:09] <wikibugs>	 6Labs, 10Horizon, 13Patch-For-Review: Horizon dashboard for managing http proxies for labs instances - https://phabricator.wikimedia.org/T129245#2152622 (10AlexMonk-WMF) Shall we still split views.py up? Or just close this as done?
[01:20:13] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys at horizon - https://phabricator.wikimedia.org/T130976#2152630 (10Luke081515)
[01:20:56] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys at horizon - https://phabricator.wikimedia.org/T130976#2152642 (10Luke081515) p:5Triage>3High
[01:22:21] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys at horizon - https://phabricator.wikimedia.org/T130976#2152630 (10Luke081515) I created a testproxy for an existing instance: Deleteing of a proxy for an existing instance does not work too.
[01:22:53] <Luke081515>	 Krenair: If you're doing horizon stuff now, maybe you can take a look at this ^ too?
[01:27:59] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys at horizon - https://phabricator.wikimedia.org/T130976#2152630 (10AlexMonk-WMF) Have you tested this against a proxy under a domain your project owns as well as direct subdomains of wmflabs.org?
[01:29:33] <Luke081515>	 Krenair: Do you mean like <name>.<project>.wmflabs.org?
[01:29:42] <Krenair>	 yes
[01:29:47] <Luke081515>	 wait a moment
[01:29:56] <Krenair>	 there is a big difference between those and <name>.wmflabs.org proxies
[01:30:20] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys at horizon - https://phabricator.wikimedia.org/T130976#2152648 (10Luke081515) Deleting of a <name>.<project>.wmflabs.org proxy works.
[01:30:22] <Luke081515>	 Krenair: ^
[01:33:32] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976#2152679 (10AlexMonk-WMF)
[01:34:55] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976#2152630 (10AlexMonk-WMF) I was going to test this on labtesthorizon but it appears that's now broken :(
[02:01:33] <wikibugs>	 6Labs, 6Operations: Labtest designate giving out Forbidden exceptions when trying to list domains - https://phabricator.wikimedia.org/T130979#2152683 (10AlexMonk-WMF)
[02:02:29] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976#2152696 (10AlexMonk-WMF) labtest issue -> T130979. Anyway, confirmed in real horizon by creating `T130976.wmflabs.org.` under andrewtestproject and then not being...
[03:22:49] <Cyberpower678>	 Niharika, Cyberbot is already logging every page it modifies.
[03:22:57] <Cyberpower678>	 It's in place.
[03:23:49] <Niharika>	 Cyberpower678: Yep, but it doesn't send in the number of links not fixed right now. 
[03:23:59] <Cyberpower678>	 Yes it
[03:24:00] <Cyberpower678>	 does
[03:24:12] <Niharika>	 Cyberpower678: Oh, okay. I see 0s only. 
[03:24:44] <Cyberpower678>	 I assume you want dead links Cyberbot can't fix rather than a number of links that remained untouched.
[03:24:54] <Cyberpower678>	 The latter seems pointless
[03:25:36] <Niharika>	 Cyberpower678: Yeah, that was the original thought but it seemed it'd be harder to compute that. Is it possible?
[03:26:11] <Cyberpower678>	  Of course.  Cyberbot counts the links it can't fix or must tag as dead.
[03:26:54] <Cyberpower678>	 At least it should.
[03:27:12] <Niharika>	 Cyberpower678: Ah, okay. How hard is it to send in that number to the logger?
[03:27:28] <Niharika>	 Cyberpower678: Can I help with that task?
[03:27:33] <Cyberpower678>	 Not hard.  It's sending it.
[03:28:16] <Niharika>	 Cyberpower678: Ahhh, okay! Thanks!
[03:28:18] <Cyberpower678>	 I recently changed the bot's configuration to run on all of wikipedia now instead of just tagged pages, so it's probably doing source formatting fixes for articles now.
[03:29:08] <Cyberpower678>	 I did that so it can cache links from other articles into the DB, so the runs are faster in the long run when we activate the checkIfDead class.
[03:29:21] <Niharika>	 Cyberpower678: Okay. Nice. 
[03:30:01] <Niharika>	 Cyberpower678: I'll send in a PR for having a timeout for curl in a moment. 
[03:30:41] <Cyberpower678>	 Niharika, Is there any way to reliably detect when the bot gets blacklisted from the site it calls.
[03:31:09] <Cyberpower678>	 Because it would be pointless to continue querying those sites, as well possibly make the bot think the site is dead.
[03:31:27] <Niharika>	 Cyberpower678: In the tests? Not really. How often do you plan to run the tests though? 
[03:31:54] <Cyberpower678>	 I mean the bot's useragent conveys the importance of why it's needed, but there will be those jerks that block the bot.
[03:32:42] <Cyberpower678>	 Niharika, the idea is that when Cyberbot checks an article, it tests every link on the page unless it is marked as dead, or if the site blacklisted the bot.
[03:33:31] <Niharika>	 Cyberpower678: Ah, okay. Now I understand. The blacklisting is likely to happen if we query the same website too frequently, right?
[03:34:22] <Cyberpower678>	 Probably.  Considering Wikipedia is large, it wouldn't be too frequent, but there are articles that use the exact same URLs as other articles, so the bot will re-check when it comes across them.
[03:35:02] <Cyberpower678>	 Cyberbot's UA is Vital Source Rescuing Bot for wikipedia.org (Cyberbot II operated by Cyberpower678).  Please contact Cyberpower678 at https://en.wikipedia.org/wiki/User:Cyberpower678 for questions and concerns.
[03:35:55] <Niharika>	 Cyberpower678: I don't think the likelihood of such a blacklisting is high, but we can ask what kaldari thinks of this. 
[03:36:04] <Cyberpower678>	 I already did.
[03:36:17] <Cyberpower678>	 He's not sure how to detect being blacklisted.
[03:36:51] <Cyberpower678>	 Some sites automatically blacklist any connection when the UA contains bot
[03:37:19] <Cyberpower678>	 In those cases Cyberbot will mistake them for dead and prematurely tag those pages.
[03:38:24] <Niharika>	 Hmm. If there's a blacklist, it'll still return a 200 right? 
[03:39:57] <Cyberpower678>	 Probably not.
[03:40:16] <Cyberpower678>	 We might get a forbidden code
[03:42:42] <Cyberpower678>	 Niharika, I love how when you bing "http code returned for blacklisted bots", one of my enwiki BRFAs pop up. :p
[03:43:57] <Niharika>	 :D
[03:47:15] <Niharika>	 301 Moved Temporarily, 401 Unauthorized, 403 Forbidden, 404 Not Found, 408 Request Timeout, 429 Too Many Requests, 503 Service Unavailable
[03:49:23] <Niharika>	 How come nobody blocks the Google scapers. 
[03:49:29] <Niharika>	 scrapers*
[03:50:21] <Cyberpower678>	  Some probably do.
[03:50:43] <Cyberpower678>	 But they know it
[03:50:59] <Cyberpower678>	 's Google and want to appear on the search engine most likely.
[03:51:18] <Niharika>	 Let's pretend to be a Google bot? :P 
[03:51:38] <Cyberpower678>	 ...
[03:51:41] <Cyberpower678>	 hmm...
[03:52:08] <Cyberpower678>	 The UA does explain the bot and who to contact.
[03:52:20] <Cyberpower678>	 If people understand that i
[03:52:36] <Cyberpower678>	 t's to source Wikipedia, they should leave the connection open.
[03:52:48] <Cyberpower678>	 Especially since they know where to go to contact.
[03:53:06] <Niharika>	 Cyberpower678: That seems good enough for now. We can worry about this when we find at least one website which has blocked us. 
[03:53:37] <Cyberpower678>	 Then all we need to do is add a multiquery function and then we can push approval.
[03:53:57] <Niharika>	 Cyberpower678: Do you think it'll lend it more credibility if you say that they can contact the Community Tech team if they have a question? And linked to our team page or something?
[03:54:10] <Cyberpower678>	 Sure.
[03:54:10] <Niharika>	 Or some subpage under our team page. 
[03:54:38] <Cyberpower678>	 Where should I point it.
[03:54:57] <Niharika>	 Let's see...
[03:55:45] <Cyberpower678>	 Cyberbot currently has information on 2400159 different links stored.
[03:56:15] <Niharika>	 Cyberpower678: Wow. That's huge! Do you think we can use that to backfill the logger? Will it be useful?
[03:56:31] <Cyberpower678>	 I don't think so.
[03:56:39] <Niharika>	 Cyberpower678: How about https://www.mediawiki.org/wiki/Community_Tech ? I'll add a template to add contacts for team members after asking others. 
[03:56:48] <Niharika>	 But it does lead to individual people. 
[03:56:51] <Cyberpower678>	 It only has link information, not what was done to the links and when.
[03:57:35] <Niharika>	 Cyberpower678: Ah, okay. We can skip that then. How many pages do you think it fixes per day (on a good day)? :) 
[03:58:20] <Cyberpower678>	 Niharika, https://en.wikipedia.org/wiki/User:Cyberbot_II/Dead-Links_Log
[03:58:38] <Cyberpower678>	 Those runs you see there are from the restricted operation.
[03:59:04] <Cyberpower678>	 That is when Cyberbot was confined to operating on only articles that have a dead tag on them.
[03:59:17] <Cyberpower678>	 I just today, released that restriction.
[03:59:41] <Cyberpower678>	 So we will see how long it takes to get across 5 million articles.
[03:59:50] <Cyberpower678>	 On the first run. :p
[04:00:39] <Niharika>	 Cyberpower678: Ah. Okay. :) Is there an improvement in how frequently the bot goes down? I haven't seen in go down in a few days, I think.
[04:01:07] <Cyberpower678>	 It
[04:01:09] <Cyberpower678>	 It
[04:01:20] <Cyberpower678>	 It's only downtime is when I installed updates.
[04:01:40] <Niharika>	 Ah, okay, that explains it. 
[04:01:46] <Cyberpower678>	 The bot is holding stable with <100 MB RAM usage
[04:02:26] <Cyberpower678>	 Niharika, you could say that I'm part of the Community Tech Team. :p
[04:03:00] <Niharika>	 Cyberpower678: You are!! :D
[04:03:34] <Niharika>	 Cyberpower678: There are a lot of awesome volunteers like you who are more a part of Community Tech than we are. :) 
[05:10:56] <SMalyshev>	 hey, what happened to "Manage Web Proxies" link on wikitech? Looks like it's disappeared
[06:17:46] <shinken-wm>	 RECOVERY - Puppet run on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0]
[06:56:27] <shinken-wm>	 RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[06:56:28] <shinken-wm>	 PROBLEM - Puppet staleness on tools-docker-builder-01 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [43200.0]
[07:50:00] <wikibugs>	 6Labs: Two factor authentication enabled unexpectedly itself on Wikitech - https://phabricator.wikimedia.org/T130953#2152827 (10Dereckson)
[07:50:11] <wikibugs>	 6Labs: Two factor authentication enabled unexpectedly itself on Wikitech - https://phabricator.wikimedia.org/T130953#2151905 (10Dereckson)
[07:50:34] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Two factor authentication enabled unexpectedly itself on Wikitech - https://phabricator.wikimedia.org/T130953#2151905 (10Dereckson)
[07:53:50] <zhuyifei1999_>	 SMalyshev: https://lists.wikimedia.org/pipermail/labs-l/2016-March/004395.html
[07:57:46] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2152833 (10Dereckson)
[07:57:51] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Two factor authentication enabled unexpectedly itself on Wikitech - https://phabricator.wikimedia.org/T130953#2152832 (10Dereckson)
[07:58:22] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Two factor authentication  unexpectedly enabled itself for Gifti - https://phabricator.wikimedia.org/T130953#2151905 (10Dereckson)
[08:06:40] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 6Operations, 10Packaging: Update phantomjs to 2.1.1 on trusty - https://phabricator.wikimedia.org/T130940#2152835 (10Dereckson)
[08:06:57] <wikibugs>	 6Labs, 10Continuous-Integration-Infrastructure, 6Operations, 10Packaging: Update phantomjs to 2.1.1 on trusty - https://phabricator.wikimedia.org/T130940#2151512 (10Dereckson) We would need a custom Debian package, as Ubuntu provides < Xenial 1.9
[08:30:02] <shinken-wm>	 PROBLEM - SSH on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:34:52] <shinken-wm>	 RECOVERY - SSH on tools-webgrid-lighttpd-1401 is OK: SSH OK - OpenSSH_6.9p1 Ubuntu-2~trusty1 (protocol 2.0)
[09:38:07] <shinken-wm>	 PROBLEM - Puppet staleness on tools-bastion-05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0]
[13:14:58] <wikibugs>	 6Labs, 10Tool-Labs: HTTO500 error on tools.wmflabs.org - https://phabricator.wikimedia.org/T130989#2152967 (10Ziad)
[13:27:13] <wikibugs>	 6Labs, 10Tool-Labs, 10wikitech.wikimedia.org: https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request HTTP/500's - https://phabricator.wikimedia.org/T130989#2152985 (10valhallasw)
[13:56:37] <wikibugs>	 6Labs, 10Tool-Labs, 10wikitech.wikimedia.org: https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request HTTP/500's - https://phabricator.wikimedia.org/T130989#2152967 (10Krenair) No, as I said on labs-l, this is because of https://gerrit.wikimedia.org/r/#/c/278909/  There's a patch but it nee...
[14:45:53] <revi>	 am I only one refused login becuase no 2FA (while I have never set up 2FA?)
[14:52:54] <tom29739>	 revi, I was yesterday.
[14:52:58] <revi>	 hmm
[14:53:05] <revi>	 is there a bug?
[14:53:41] <revi>	 found it, https://phabricator.wikimedia.org/T130892
[14:55:09] <revi>	 how dare 2FA is required while I never set it up :(
[14:55:14] <tom29739>	 revi, all by itself? You didn't even go into the 'enable two-factor authentication' page accidentally?
[14:55:32] <revi>	 I never logged in to wikitech in last 3 months
[14:55:40] <revi>	 I don't recall for older dates
[14:57:06] <revi>	 so stupid mediawiki is thinking I enabled if I ever once visited that page?
[14:57:16] <revi>	 regardless of confirmation or not?
[14:59:33] <tom29739>	 It did it for me without confirmation when I visited that page and then went back.
[14:59:40] <tom29739>	 No confirmation needed.
[14:59:48] <revi>	 what the fuck.
[15:00:08] <revi>	 so I have to recall from 2013 if I have clicked or not
[15:00:14] <revi>	 I'll just file a bug :P
[15:01:27] <revi>	 project labs/wikitech.wikimedia.org should be sufficient I hope
[15:01:35] <revi>	 labs & wikitech
[15:01:39] <tom29739>	 That's what I did.
[15:01:53] <tom29739>	 andrewbogott reset it for me in here.
[15:02:00] <revi>	 good
[15:02:51] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Reset 2FA on wikitech for User:Revi and User:Hym411 - https://phabricator.wikimedia.org/T130994#2153079 (10revi)
[15:03:39] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2153091 (10revi)
[15:03:41] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Reset 2FA on wikitech for User:Revi and User:Hym411 - https://phabricator.wikimedia.org/T130994#2153079 (10revi)
[15:26:00] <halfak>	 o/ yuvipanda 
[15:26:04] <halfak>	 any chance you are around?
[15:26:26] <halfak>	 Got a ping from Sean re. PAWS workshop in Chicago
[15:36:23] <tom29739>	 Looks like Special:FormEdit doesn't work full-stop for any form on Wikitech.
[15:43:00] <wikibugs>	 6Labs, 6Operations: Labtest designate giving out Forbidden exceptions when trying to list domains - https://phabricator.wikimedia.org/T130979#2152683 (10Andrew) I see the same behavior you're seeing if I visit the DNS panel for a project that I'm a 'user' in but not a 'projectadmin' in.  If I am a projectadmin...
[15:46:34] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976#2153149 (10Andrew) I can reproduce this problem in production but not on labtest.
[16:09:48] <Luke081515>	 andrew: at which domain is the horizon for labstest?
[16:13:19] <Luke081515>	 andrewbogott: ^
[16:13:27] * Luke081515 sometimes mixes up phabricator & IRC names
[16:14:16] <andrewbogott>	 it's at labtesthorizon.wikimedia.org but you don't have access there, just dev accounts
[16:14:26] <Luke081515>	 ah, ok
[16:21:16] <valhallasw`cloud>	 Luke081515: re: arc PATH export: add it to your ~/.profile
[16:21:28] <valhallasw`cloud>	 (but you maybe already got a response)
[16:21:37] <wikibugs>	 6Labs, 10Tool-Labs, 10wikitech.wikimedia.org, 13Patch-For-Review, 5WMF-deploy-2016-03-22_(1.27.0-wmf.18): https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request HTTP/500's - https://phabricator.wikimedia.org/T130989#2153201 (10Krenair) 5Open>3Resolved a:3Krenair `16:21:14 Synchro...
[16:21:42] <Luke081515>	 no, not yet. Thanks, valhallasw :)
[16:22:01] <valhallasw`cloud>	 Krenair: thanks!
[16:22:12] <valhallasw`cloud>	 I had missed your comment on labs-l
[16:24:33] <Krenair>	 andrewbogott, I don't have access to californium - are you sure the credentials are set up properly in production horizon?
[16:24:47] <Luke081515>	 valhallasw`cloud: Thanks, that solved my problem :)
[16:25:01] <andrewbogott>	 that's what I'm looking at now.  They were slightly different but making them the same didn't help
[16:39:51] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976#2153210 (10Andrew) The issue is in this code block:              for recordset in c.recordsets.list(zoneid):                 lookingfor = record + '.' + domain...
[16:44:51] <Luke081515>	 !log rcm.cac updating all git repos
[16:44:54] <labs-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm.cac/SAL, Master
[16:58:17] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon, 13Patch-For-Review: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976#2153241 (10Andrew) 5Open>3Resolved a:3Andrew fixed by attached patch
[17:16:22] <Luke081515>	 andrewbogott: Is you patch for T130976 already live?
[17:16:23] <stashbot>	 T130976: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976
[17:17:44] * Luke081515 is still unable to delete his old proxys
[17:18:40] <andrewbogott>	 Luke081515: yes
[17:18:47] <andrewbogott>	 oh, really?
[17:19:13] <Luke081515>	 Error: Unable to delete proxy: login-rcm-wiki-ca.wmflabs.org
[17:20:10] <andrewbogott>	 Luke081515, try again?
[17:20:17] <andrewbogott>	 I'm watching the logs this time
[17:21:03] <Luke081515>	 I need a moment, my handy is hanging
[17:21:08] <Luke081515>	 I need it for 2FA...
[17:21:44] <Luke081515>	 andrewbogott: Ok, I will try it again now
[17:21:59] <Luke081515>	 done
[17:22:19] <andrewbogott>	 WARNING:wikimediaproxydashboard.views:Woops! Failed domain ID for domain wmflabs.org
[17:22:35] <andrewbogott>	 Action (u'deleted proxy', u'login-rcm-wiki-ca.wmflabs.org') Failed for No domain ID
[17:23:30] <Luke081515>	 maybe caused by the switch to horizon? I can also try it with a proxy I created inside horizon, if you want
[17:23:47] <andrewbogott>	 Krenair: that's happening because it's getting wmflabs.org and not wmflabs.org.
[17:24:05] <Krenair>	 ugh
[17:24:08] <andrewbogott>	 I hope that's not because my import script gave everything the wrong name...
[17:24:24] <Krenair>	 probably an inconsistency between labtest and labs
[17:24:43] <andrewbogott>	 actually, is that domain name is coming from the proxy api isn't it?
[17:24:57] <Krenair>	 oh, right. probably, yes
[17:25:00] <Krenair>	 ugh
[17:25:15] <Luke081515>	 andrewbogott: Maybe this helps you: I tried to delete a proxy I created inside horizon, that works
[17:25:21] <andrewbogott>	 We can just add "If this doesn't end in a dot add a dot" 
[17:25:26] <andrewbogott>	 Luke081515: yeah, that fits
[17:25:35] <Luke081515>	 the difference: for example the domain login-rcm-wiki-ca.wmflabs.org shows as backend http://rcm-wiki-ca.rcm.eqiad.wmflabs:8080 
[17:25:43] <Krenair>	 can't we just fix all the existing data?
[17:25:46] <andrewbogott>	 Krenair: so proxies that predate this switchover won't have the dot
[17:25:55] <Luke081515>	 if I would create a proxy in horizon it just would show: rcm-wiki-ca:8080
[17:26:18] <andrewbogott>	 Krenair: do you have access to the proxy db?
[17:26:27] <Krenair>	 yes
[17:26:42] <andrewbogott>	 That fix would just be a bit of sql magic, but I don't especially trust myself with that
[17:27:25] <Krenair>	 I'll take a backup before doing anything dangerous
[17:31:05] <Luke081515>	 Krenair: You can ping me, if I should try it again (delete my proxys)
[17:40:00] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon, 13Patch-For-Review: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976#2153266 (10Luke081515) 5Resolved>3Open a:5Andrew>3AlexMonk-WMF Needs some data-schemata change...
[17:40:08] <Luke081515>	 (for other affected users)
[17:54:22] <Krenair>	 sqlite> select * from route where domain like '%.';
[17:54:24] <Krenair>	 972|T130976.wmflabs.org.|155
[17:54:24] <stashbot>	 T130976: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976
[17:54:26] <Krenair>	 973|geotest.wmflabs.org.|81
[17:57:10] <Krenair>	 Luke081515, try now
[17:57:24] <Luke081515>	 ok, wait a moment
[17:57:38] <Luke081515>	 Krenair: Works now, thanks
[17:57:43] <Krenair>	 I updated login-rcm-wiki-ca.wmflabs.org.
[17:58:00] <Luke081515>	 Yeah, I deleted this proxy now
[17:58:09] <Luke081515>	 this works
[17:58:40] <Krenair>	 will just test this out and then apply it to the rest
[18:01:06] <Krenair>	 sqlite> update route set domain = printf('%s.', domain) where domain not like '%.';
[18:01:06] <Krenair>	 sqlite>
[18:03:22] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon, 13Patch-For-Review: Unable to delete proxys directly under wmflabs.org at horizon - https://phabricator.wikimedia.org/T130976#2153271 (10AlexMonk-WMF) 5Open>3Resolved Took a backup of `/etc/dynamicproxy-api/data.db` at `/root/data.db.2016-03-26.bak` on novaproxy...
[18:53:10] <Luke081515>	 Krenair: Can you take a look at the proxy DB again? Now I'm unable to delete the ubstest.rcm.wmflabs.org proxy :-/
[18:53:24] <Krenair>	 what error do you get?
[18:53:37] <Luke081515>	 Error: Unable to delete proxy: ubstest.rcm.wmflabs.org.
[18:54:01] <Krenair>	 sqlite> select * from route where domain = 'ubstest.rcm.wmflabs.org.';
[18:54:01] <Krenair>	 965|ubstest.rcm.wmflabs.org.|128
[18:54:18] <Krenair>	 project rcm
[18:55:03] <Krenair>	 backend http://ubstest.rcm.eqiad.wmflabs:8080
[18:56:37] <Krenair>	 Luke081515, can you look into the DNS info and tell me what IP that domain is pointing to?
[18:56:41] <Krenair>	 we want the A record
[18:57:16] <Luke081515>	 Krenair: You mean the IP of the instance? this is 10.68.16.85
[18:57:23] <Luke081515>	 currently the instance is paused
[18:57:24] <Krenair>	 no
[18:57:39] <Krenair>	 I mean the IP that the public DNS record points to
[18:57:56] <Luke081515>	 I don't know how to get this, actually
[18:59:29] <Luke081515>	 Krenair: How can I find that out?
[19:00:13] <Krenair>	 under Web Proxies in the sidebar
[19:00:22] <Krenair>	 there should be a Domains link to https://labtesthorizon.wikimedia.org/project/dns_domains/
[19:00:24] <Krenair>	 but without the labtest bit
[19:00:35] <Krenair>	 click the link
[19:00:51] <Krenair>	 you should see a table of domains for your project
[19:00:59] <Luke081515>	 there is TTL and a Serial and I mail
[19:01:20] <Luke081515>	 *a mail
[19:01:29] <Krenair>	 for rcm.wmflabs.org ?
[19:03:01] <Luke081515>	 yep
[19:03:08] <Luke081515>	 TTL is 120
[19:03:16] <Krenair>	 click manage records
[19:03:23] <Krenair>	 for that specific row in the table
[19:03:38] <Krenair>	 you should get a new table containing subdomains of rcm.wmflabs.org
[19:03:43] <Luke081515>	 then I have two, one with typ = NS and one with typ = SOA
[19:03:45] <Krenair>	 one of them will be ubstest.rcm.wmflabs.org
[19:03:52] <Krenair>	 no type As?
[19:04:06] <Luke081515>	 no, there are just two times rcm.wmflabs.org and no AS
[19:04:22] <Krenair>	 sounds like something went wrong with the migration script
[19:04:24] <Krenair>	 andrewbogott?
[19:05:11] <andrewbogott>	 that's a whole lot of scrollback… what are you expecting to see?
[19:06:09] <Krenair>	 an A record pointing to the proxy
[19:06:46] <andrewbogott>	 for ubstest.rcm.wmflabs.org ?
[19:07:15] <andrewbogott>	 (I have to run — open a bug and assign it to me and I'll look)
[19:21:59] <Krenair>	 Luke081515, ^
[19:22:14] <Luke081515>	 ok
[19:22:39] <Krenair>	 andrewbogott, this is another domain that fails on labs-ns2
[19:22:51] <Krenair>	 in addition to the one I posted in the other channel yesterday
[19:23:37] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Can't delete the proxy ubstest.rcm.wmflabs.org - https://phabricator.wikimedia.org/T131004#2153335 (10Luke081515)
[19:23:46] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Can't delete the proxy ubstest.rcm.wmflabs.org - https://phabricator.wikimedia.org/T131004#2153347 (10Luke081515) a:3Andrew
[19:25:56] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Can't delete the proxy ubstest.rcm.wmflabs.org - https://phabricator.wikimedia.org/T131004#2153367 (10Luke081515) (It's not urgent, so don't hurry)
[20:44:47] <Cyberpower678>	 kaldari, Niharika: the logging interface seems to be bugged up./
[20:44:59] <Cyberpower678>	 The table is mashed up on the right hand side.
[20:49:07] <Niharika>	 Cyberpower678: I'm going to bed right now but it seems okay on phone. A screenshot would be useful.
[20:49:34] <Cyberpower678>	 Niharika, good night
[20:49:45] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Can't delete the proxy ubstest.rcm.wmflabs.org - https://phabricator.wikimedia.org/T131004#2153397 (10Andrew) The problem with this record is that it has hostname 'ubstest.rcm' under domain 'wmflabs.org' rather than having hostname 'ubstest' under domain 'rcm.wmflabs....
[20:49:55] <Niharika>	 Have a good day, Cyberpower678.
[20:53:40] <wikibugs>	 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2153399 (10csteipp) I've gone ahead and disabled 2FA on wikitech for users who had is_validated = 0, length(scratch_tokens)=84, and a user_...
[21:10:59] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Reset 2FA on wikitech for User:Revi and User:Hym411 - https://phabricator.wikimedia.org/T130994#2153079 (10JanZerebecki) T130892#2153399 should have fixed this for you, please verify.
[21:11:05] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Two factor authentication  unexpectedly enabled itself for Gifti - https://phabricator.wikimedia.org/T130953#2151905 (10JanZerebecki) T130892#2153399 should have fixed this for you, please verify.
[21:15:06] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Two factor authentication  unexpectedly enabled itself for Gifti - https://phabricator.wikimedia.org/T130953#2153438 (10csteipp) 5Open>3Resolved a:3csteipp Confirmed, Gifti was one of the users I removed 2FA for.
[21:20:20] <wikibugs>	 6Labs, 10wikitech.wikimedia.org: Reset 2FA on wikitech for User:Revi and User:Hym411 - https://phabricator.wikimedia.org/T130994#2153079 (10csteipp) 2FA for Revi was removed. Hym411 has a user_touched value from this past week. @revi, is 'Hym411' your account? If so, can you point to any places where that rela...
[21:22:59] <wikibugs>	 6Labs, 10Tool-Labs, 6Community-Tech-Tool-Labs, 10MediaWiki-Vagrant, and 2 others: Tools, Labs, & MediaWiki-Vagrant help desk (#wikimedia-hackathon-2016) - https://phabricator.wikimedia.org/T130780#2153444 (10valhallasw) >>! In T130780#2146412, @bd808 wrote: > Hoping that @valhallasw, @Reedy and some others...
[21:46:07] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Can't delete the proxy ubstest.rcm.wmflabs.org - https://phabricator.wikimedia.org/T131004#2153451 (10Andrew) OK, I added some hardcoded domains to the import script (ex: rcm.wmflabs.org) and re-imported.  @Luke081515 does this work reasonably now?
[21:50:07] <wikibugs>	 6Labs, 10Labs-Infrastructure, 10Horizon: Can't delete the proxy ubstest.rcm.wmflabs.org - https://phabricator.wikimedia.org/T131004#2153465 (10Luke081515) 5Open>3Resolved   Success: Deleted Proxy: ubstest.rcm.wmflabs.org. Thanks @Andrew
[22:33:40] <wikibugs>	 6Labs, 6Operations: Labtest designate giving out Forbidden exceptions when trying to list domains - https://phabricator.wikimedia.org/T130979#2153502 (10Andrew) ok, actually preventing the tab from showing up for non-projectadmins turns out to be hard (the code doesn't do policy checks properly).  And, anyway,...