[01:38:13] andrewbogott: hey, if you're around. How I can add a role to an instance using horizon? I was doing it via special:NovaInstance (configure option) [01:38:57] Amir1: Why not using wikitech? [01:39:22] nova instance doesn't show anything to me [01:39:47] did you select the project? [01:39:52] yeah [01:39:57] I think it's the old bug [01:40:04] I don't know that bug [01:40:04] let me log out and then login [01:40:41] sometimes novainstance doesn't show anything, in that case you need to log out and login [01:41:18] I think in horizon that thing with pupple is currently not possible [01:41:50] yes, it's fixed now [01:46:01] I think that's only possible via wikitech [01:46:06] and yes, that is a known bug [01:46:14] (the empty list thing) [03:36:34] 6Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2166923 (10Sigma) So the view query isn't indexed or what? I don't actually know how Labs works so that probably doesn't make sense, but I'm interested to know why everything is slow. [06:52:18] I am not sure why https://de.wikipedia.org/wiki/Hooded-Sittich appeas in my namespace. This redirect should be deleted as no german name. [06:52:45] But why as well in my namespace? Bug? [07:00:40] Results (Found 3): morebots, labs-morebots, hyperon, [07:00:40] @search admin [07:00:48] Results (Found 5): help, demon, deployment-beta-docs-1, bible, doc, [07:00:48] @search doc [07:01:18] Results (Found 23): morebots, labs-morebots, stats, add, tools-bug, status, tools-web, pastebin, awstats, filemoves.py, toolsbeta, tdb, logs, redis, newlabs-rl, newlabs2, accessproblems, newlabs, tools-equiad, relay, hook, tools-status, precise, [07:01:18] @search tools [08:20:15] 10Tool-Labs-tools-Other, 10Wikimedia-Hackathon-2016: Discuss the future of Commonists Org metrics tools - https://phabricator.wikimedia.org/T131438#2167261 (10JeanFred) [08:36:10] valhallasw`cloud: I've got a WTF on wikitech. The login form isn't presenting me with an 2fa prompt [08:36:35] oh... never mind. wtf happend to the long form? [08:38:01] oh, fun. My 2FA has been reset? [08:38:20] I think wikitech 2fa is busted [08:38:28] my 2fa works on horizon [08:38:45] I wonder what was in the new branch? [08:40:11] I'm betting 1.27.0-wmf.19 may have busted wikitech [08:41:02] bd808: I'm going to try and re-set 2fa and hope I don't lock myself out completely [08:41:42] valhallasw`cloud: don't yet [08:42:14] I think there is a bug in the new mw version [08:42:31] reedy is going to roll wikitech back [08:47:54] 6Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2167336 (10jcrespo) MySQL views cannot have indexes (they are not materialized), the underlying table have. However, the views also perform sanitization of private fields, which requires extra filtering and may... [08:49:10] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2167379 (10Reedy) [08:49:45] Error: RecordSet belongs in a child zone: librarybase.wmflabs.org. [08:49:48] dawut [08:50:01] is this the DNS zoning stuff that was discussed yesterday? [08:53:25] 6Labs, 10DNS, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2167425 (10valhallasw) [08:55:33] 6Labs, 10DBA, 6Operations: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2167442 (10jcrespo) The following tables on enwiki have been already imported: ``` 73350868 logging 51366344 archive 68959980 article_feedback 49353723 updates 39641126 page 33369615 article_feedback_revisions... [09:04:56] bd808: where are the logs for http://reading-web-staging.wmflabs.org/ ? [09:05:01] i'm trying to work out why it's down [09:05:11] sorry if that's a stupid question :) [09:09:48] jdlrobson: http://horizon.wikimedia.org/ , choose project reading-web-staging, click DNS on the left, click Proxies [09:09:50] thta should list them [09:10:38] and hopefully tells you which host it forwards to [09:14:50] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2167612 (10Parent5446) @Reedy I cannot seem to reproduce this locally. Could you provide some reproduction steps? I've tried visiting Special:Userrights and other res... [09:15:27] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2167617 (10Reedy) >>! In T131445#2167612, @Parent5446 wrote: > @Reedy I cannot seem to reproduce this locally. Could you provide some reproduction steps? I've tried v... [09:16:11] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2167618 (10Parent5446) @Reedy Just so I know all the details, were you logged in already? And I presume your account has 2FA enabled on it? [09:17:22] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2167619 (10Reedy) >>! In T131445#2167618, @Parent5446 wrote: > @Reedy Just so I know all the details, were you logged in already? And I presume your account has 2FA e... [09:23:12] i had my gerrit user added to the tools project, but i still can't seem to ssh in. From the docs it looks like i should ssh to maybe tools-login.wmflabs.org, but it keeps rejecting me with permission denied (same account that works with other labs projects) [09:23:16] any ideas? [09:23:59] s/gerrit/wikitech/ [09:24:32] ebernhardson: let me take a look [09:25:02] ebernhardson: you're using lowercase ebernhardson to login? [09:25:06] valhallasw`cloud: yes [09:25:15] oh, does that make a difference here? lemme try [09:25:51] Apr 1 09:21:53 tools-bastion-05 sshd[25214]: Failed publickey for ebernhardson from XX.YY.ZZ.AA port 49065 ssh2: RSA SHA256:.... [09:25:54] hrm [09:26:05] oh, sec i might be getting the wrong key... [09:26:48] yes, the problem was complete PEBKAC. my ~/.ssh/config sets the right key for most wmf services, but tools isn't using the regular bastions... [09:27:34] valhallasw`cloud: thanks for looking, was all my own brokenness :) [09:29:49] 10Tool-Labs-tools-Other, 10Wikimedia-Hackathon-2016: Discuss the future of Commonists Org metrics tools - https://phabricator.wikimedia.org/T131438#2167667 (10JeanFred) @Abit : You mentionned you were interested in having a look at such tools. I’d be happy to hear your thoughts about these :) [09:31:04] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2167669 (10Reedy) And the error message was: ``` "oathauth-abortlogin": "The two-factor authentication token provided was invalid.", ``` [09:33:13] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2167674 (10Parent5446) Literally the only place that error message is used is in the AbortChangePassword hook... I think this might be due to https://gerrit.wikimedi... [09:42:40] 10Tool-Labs-tools-Other, 10Wikimedia-Hackathon-2016: Discuss the future of Commonists Org metrics tools - https://phabricator.wikimedia.org/T131438#2167686 (10Abit) @JeanFred Tried to find you but got waylaid. I'm in the Jerusalem room, where are you? [09:45:47] 10Tool-Labs-tools-Other, 10Wikimedia-Hackathon-2016: Discuss the future of Commonists Org metrics tools - https://phabricator.wikimedia.org/T131438#2167690 (10Lena) When are you planning to have this discussion ? [09:46:54] 6Labs, 10Spam: Add default spam prevention tools or settings to wmflabs instances - https://phabricator.wikimedia.org/T131459#2167691 (10Quiddity) [09:48:20] (03PS1) 10Merlijn van Deen: Log hackathon tasks to #wmhack [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/280841 [09:48:36] 6Labs, 10Spam: Add default spam prevention tools or settings to wmflabs instances - https://phabricator.wikimedia.org/T131459#2167704 (10Quiddity) [09:49:18] (03CR) 10Merlijn van Deen: [C: 032] Log hackathon tasks to #wmhack [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/280841 (owner: 10Merlijn van Deen) [09:52:00] 10Tool-Labs-tools-Other, 10Wikimedia-Hackathon-2016: Discuss the future of Commonists Org metrics tools - https://phabricator.wikimedia.org/T131438#2167713 (10JeanFred) >>! In T131438#2167686, @Abit wrote: > @JeanFred Tried to find you but got waylaid. I'm in the Jerusalem room, > where are you? In the Masad... [09:54:21] (03CR) 10Merlijn van Deen: [V: 032] "jenkins y u so slow" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/280841 (owner: 10Merlijn van Deen) [09:54:37] !log tools.wikibugs Updated channels.yaml to: a1754bac96c8fcb14532b808d5ca9db15e8f3c25 Merge "Log hackathon tasks to #wmhack" [09:54:37] 10Tool-Labs-tools-Other, 10Wikimedia-Hackathon-2016: Discuss the future of Commonists Org metrics tools - https://phabricator.wikimedia.org/T131438#2167720 (10JeanFred) >>! In T131438#2167690, @Lena wrote: > When are you planning to have this discussion ? I had not really planned to work on these things at th... [09:54:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [09:58:47] 6Labs, 10MediaWiki-Vagrant, 10MediaWiki-extensions-Newsletter: Cannot enable/list roles in newsletter-test instance - https://phabricator.wikimedia.org/T131460#2167726 (1001tonythomas) [09:59:36] 6Labs, 10MediaWiki-Vagrant, 10MediaWiki-extensions-Newsletter: Cannot enable/list roles in newsletter-test instance - https://phabricator.wikimedia.org/T131460#2167738 (1001tonythomas) @Qgil found that the instance do not run echo, and this task blocks us from enabling echo :( [10:00:14] 10Tool-Labs-tools-Other, 10Wikimedia-Hackathon-2016: Discuss the future of Commonists Org metrics tools - https://phabricator.wikimedia.org/T131438#2167740 (10JeanFred) >>! In T131438#2167690, @Lena wrote: > When are you planning to have this discussion ? I had not really planned to work on these things at th... [10:01:11] 10Tool-Labs-tools-Other, 10Wikimedia-Hackathon-2016: Present and discuss the Commonists Org program evaluation tools to WMF stakeholders - https://phabricator.wikimedia.org/T131438#2167745 (10JeanFred) [10:09:39] 6Labs, 10Labs-Infrastructure, 7Graphite: Graphite is unable to detect, if a paused instance is resumed - https://phabricator.wikimedia.org/T131022#2153858 (10fgiunchedi) was the resumed instance accessible after the resume? can you reproduce it? it'd be interesting to log in into the instance and see what st... [10:31:56] https://www.irccloud.com/pastebin/nVFDdBPz/ [10:32:13] wrong win [10:41:32] Hi, anyone around could help me about why my under-minute queries are getting killed in tool-labs? [11:04:56] The web server associated with my labs instance seems to be down. I can't get to http://librarybase.wmflabs.org even though the instance (librarybase-reston-01) is up and I rebooted apache just now with no incident. [11:13:47] 6Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2168017 (10jcrespo) Please check that this is intended (should mysql be installed by labs::dns, or is it independent?) https://puppet-compiler.wmflabs.org/2270/ Basically, please review ht... [11:15:27] I don't think it's a Labs problem because my other instance is just fine. [11:16:48] harej: DNS stuff, tried to fix with tarrow , try librarybase-web.wmflabs.orf [11:17:03] http://librarybase-web.wmflabs.org/wiki/Librarybase:Home works [11:17:14] hopefully librarybase.wmflabs works :| [11:17:17] eventually [11:17:50] https://phabricator.wikimedia.org/T131448#2167425 [11:18:35] valhallasw`cloud: where are you sitting now? [11:18:55] 6Labs, 10DNS, 6Operations, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168029 (10Harej) Stating for the record that `http://librarybase.wmflabs.org` worked just fine until today. [11:19:26] 6Labs, 10DNS, 6Operations, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168030 (10Harej) p:5Triage>3Unbreak! [11:20:19] Ehm... no clue about my problem? [11:22:12] jem: is mysql getting killed or just the query? [11:24:30] bd808: My tests seem to indicate that it's killed when using php functions, but not nysql from shell [11:24:32] 6Labs, 10DNS, 6Operations, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168047 (10Dereckson) @yuvipanda Is the priority of this task correctly asserted? [11:24:34] The exact same query [11:24:45] mysql* [11:25:32] timeouts? [11:25:50] I just see "Killed" from the shell [11:25:53] hmmm... I think chase enabled some strict limits on process runtime on tools-login.wmflabs.org (tools-bastion-05.tools.eqiad.wmflabs). If you are running there that may be a cause [11:26:09] 6Labs, 10DNS, 6Operations, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168049 (10Harej) I consider not being able to access Librarybase to be a production outage, the fact that is currently accessible from an alternate URL no... [11:26:38] Yes [11:26:45] bd808: do those limits apply on other labs instances as well? [11:26:48] Last one after 23 seconds [11:26:58] jem: try using tools-dev.wmflabs.org instead [11:27:16] we are trying some new limits out to keep tools-login more responsive [11:27:30] harej: I think the limits are only on tools-login [11:27:36] Ok, I guess I can access the result from outside using ssh also [11:27:55] bd808: interesting. i have some scripts on one of my labs instances that have been getting killed. i assume because it's asking for more ram than is assigned to the instance? [11:28:23] the fact that it's *killed* is interesting to me; i thought it would get out-of-memory errors instead. [11:29:01] harej: possibly. You may be getting hit by http://linux-mm.org/OOM_Killer [11:29:12] ah. so it kills the script before it could error out [11:30:14] 6Labs, 10DNS, 6Operations, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168050 (10Harej) [11:30:36] there should be logs in /var/log/syslog if OOMKiller is doing it [11:35:11] Yes, I can see my kills in there (wdb.php) [11:38:30] bd808: outside in front, but coming back inside in a bit [11:40:17] valhallasw`cloud: cool. I'm at a table in hacking rooms. Not sure which room number. maybe 1.5? [11:42:30] jem: problemas? [11:46:38] Hombre, mafk :) [11:47:17] Pues voy a probar ahora, es que por esas consultas matadas no se estaban actualizando los rankings de eswiki [11:47:43] 6Labs, 10DNS, 6Operations, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168147 (10valhallasw) p:5Unbreak!>3Normal I'm sorry @harej, but given that this is a) labs (and therefore by definition not production) and b) there i... [11:48:51] !log newsletter Joined project as admin to debug T131460 [11:48:52] T131460: Cannot enable/list roles in newsletter-test instance - https://phabricator.wikimedia.org/T131460 [11:48:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Newsletter/SAL, Master [11:49:14] jem: suerte pues, yo te acabo de inundar CAT:BR con el bot de redirecciones rotas :P [11:50:09] Vaya :) [11:50:56] Pues no sale ninguna, parece que había alguien rápido al tanto [11:51:51] Krenair: if you're around, could you take a look at https://phabricator.wikimedia.org/T131448#2168147 ? I haven't quite followed the changes wrt the proxy and dns... [11:53:53] valhallasw`cloud, it doesn't resolve [11:56:06] Krenair: I think (valhallasw`cloud correct me), that you can't actually add just that domain (or www....) to the proxy [11:56:17] OK, let me update the issue so it's clearer what I tried and what does/doesn't work [11:56:22] sorry, I should have done that before [12:00:16] 6Labs, 10DNS, 6Operations, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168165 (10valhallasw) [12:00:18] am looking into designate now [12:00:40] Krenair: it did not resolve in the initial situation (even though there was an entry in the proxy list) -- I then deleted the entry, but I could not re-add it afterwards. [12:04:29] valhallasw`cloud, I think this is https://phabricator.wikimedia.org/T131367 [12:04:34] designateclient.exceptions.BadRequest: RecordSet belongs in a child zone: librarybase.wmflabs.org. [12:05:11] Yes, I think that's probably it [12:05:25] let's see if I can fix this there [12:06:43] valhallasw`cloud, now it works [12:07:07] \o/ [12:07:10] Krenair: thank you! [12:07:19] 6Labs, 10DNS, 6Operations, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168196 (10Krenair) 5Open>3Resolved a:3Krenair I've handled this specific case, the generic issue is T131367. Also, not a production outage, this is... [12:09:57] 6Labs, 10MediaWiki-Vagrant, 10MediaWiki-extensions-Newsletter: Cannot enable/list roles in newsletter-test instance - https://phabricator.wikimedia.org/T131460#2168239 (1001tonythomas) Fixed //magically// after : ``` 17:19 tonythomas: hmm just tried `vagrant roles list` -- let me test the provision... [12:10:48] 6Labs, 10MediaWiki-Vagrant, 10MediaWiki-extensions-Newsletter: Cannot enable/list roles in newsletter-test instance - https://phabricator.wikimedia.org/T131460#2168246 (1001tonythomas) 5Open>3Resolved a:301tonythomas [12:23:37] 6Labs: Add default spam prevention tools or settings to wmflabs instances - https://phabricator.wikimedia.org/T131459#2168336 (10Peachey88) [12:30:02] 6Labs, 10DNS, 6Operations, 10Traffic: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168396 (10BBlack) [12:30:22] 6Labs, 6Operations: librarybase project cannot create a proxy for librarybase.wmflabs.org - https://phabricator.wikimedia.org/T131448#2168398 (10BBlack) [12:37:03] (03PS1) 10Lokal Profil: Updating Commons categories for se-bbr [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280890 [12:37:45] (03PS2) 10Lokal Profil: Updating Commons categories for se-bbr [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280890 (https://phabricator.wikimedia.org/T131483) [12:38:12] 6Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: can't access nova configuration for instance integration-slave-trusty-1025 - https://phabricator.wikimedia.org/T131486#2168457 (10JanZerebecki) [12:41:34] 6Labs, 10Tool-Labs, 10Continuous-Integration-Infrastructure, 7Blocked-on-RelEng, 13Patch-For-Review: debian-glue tries to fetch obsolete package - https://phabricator.wikimedia.org/T125999#2168493 (10hashar) 5Open>3Resolved a:3hashar Yup looks fine now. [12:45:39] 6Labs, 10Labs-Infrastructure, 10wikitech.wikimedia.org: can't access nova configuration for instance integration-slave-trusty-1025 - https://phabricator.wikimedia.org/T131486#2168502 (10JanZerebecki) 5Open>3Invalid I did something wrong works now. [13:12:19] (03PS1) 10Lokal Profil: Load project from config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280903 (https://phabricator.wikimedia.org/T131344) [13:30:50] (03PS2) 10Lokal Profil: Load project from config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280903 (https://phabricator.wikimedia.org/T131344) [13:34:40] * bd808 wonders why stashbot isn't rejoining... [13:35:55] bd808: I'm going ot persist the what of thoughts on making bastion sharing more sane today there isn't much going on there really atm [13:36:32] in particular the runtimes restriction is not active [13:36:34] # Max CPU minutes that a proc is allowed [13:36:34] # i.e. if using 100% of a cpu kill after 360 minutes [13:36:35] #* hard cpu 360 [13:37:31] chasemp: hmmm [13:37:37] good to know [13:38:28] bastion-05 it half-dead, even if if cpus are around 95% idle [13:38:31] *is [13:39:23] I've been seeing the same. Maybe NFS is sad today? [13:39:41] I could see how a hackathon might put extra pressure on the Tools instances [13:40:01] NFS in general is fine atm [13:40:10] but NFS from bastion may in trouble but I dn't see it anyway [13:40:28] someone is caushing enough load to create issues but I'm not sure of the what [13:40:33] I did some mysqldump running [13:40:42] and a few other things I know to be a bad idea on that host [13:41:05] oh yeah holy crap lots of ppl there [13:43:32] wasn't labs memory raised to 1 TB last week? [13:43:37] 6Labs, 10Horizon: Proxy corner case: proxy name foo.wmflabs.org == domain name foo.wmflabs.org - https://phabricator.wikimedia.org/T131367#2165096 (10bd808) >>! In T131367#2165108, @Andrew wrote: > For the record, I'm not sure that fixing this is a better solution than just saying "Don't do that then" Would... [13:44:08] that is not related to the finite resources in the login bastion really [13:44:34] it's so hosed up now I can't diagnose [13:44:52] I can't get the stashbot grid job to start either :/ [13:45:13] chasemp: I'm in if you want to feed me commands to run via PM [13:45:30] yeah I got "Killed" "Killed" only by submitting job to sge from bastion ;( [13:45:30] "13:45:16 up 8 days, 15:36, 34 users, load average: 10.18, 7.88, 6.42" [13:45:32] load keep climbing and I'm not sure why [13:45:43] usually I can track it back to some hung IO job [13:46:57] I can login through bastion-02, if you wish that I check something for you, I'm here [13:47:44] bd808: diminishing returns on figuring out who is hosing up the host [13:47:46] I would jsut reboot atm [13:48:04] load is dropping fast actually [13:48:17] down to 3.3 now [13:48:24] maybe I did catch them [13:49:05] I'm still not getting the stashbot job to start on the grid [13:49:11] (03PS3) 10Lokal Profil: Load project from config for unused images/images without id [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280903 (https://phabricator.wikimedia.org/T131344) [13:49:19] bd808: what's the err? [13:49:44] that's the weird thing, no error but bot is not joining IRC channels [13:50:01] I may have to crank up the log level to find out why I guess [13:50:47] the grid itself seems ok [13:51:01] though I see ppl running jobs not as bot accounts and I'm not thrilled about it [13:51:01] bd808: let me strace it [13:51:07] yeah I think it more likely my bot's problem [13:51:20] select(0, NULL, NULL, NULL, {0, 200000}) = 0 (Timeout) [13:51:20] gettimeofday({1459518673, 289394}, NULL) = 0 [13:51:23] ad infinitum [13:51:54] that would be the polling loop of the irc bot I imagine [13:52:06] yeah, that sounds reasonable [13:53:01] I wonder if a bunch of folks are proxying irc via labs and we hit a limit with freenode? [13:53:07] 6Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2168806 (10Andrew) The patch and change looks fine to me. [13:53:26] bd808: each exec host has a seperate IP + identd for that exact reason [13:53:53] that^ :) [13:54:05] bd808: let me grab my stuff, I'll move over to your side of the building [13:54:26] valhallasw`cloud: bd808 how goes the "labs table" help desk initiative? [13:54:41] pretty quiet right now. [13:54:50] there was some drama earlier [13:55:02] 2fa on wikitech is broken in the latest MW release [13:55:16] Reedy rolled it back to last weeks version [13:55:26] and then some dns head scratching [13:56:29] 6Labs, 10Horizon: Proxy corner case: proxy name foo.wmflabs.org == domain name foo.wmflabs.org - https://phabricator.wikimedia.org/T131367#2168821 (10Andrew) > Would this mean that we can no longer have https protection for Horizon proxied services? I was thinking more in terms of the opposite case: 'If you... [13:58:07] bd808: I'm here now and have a few hours if there are any helpdesk puzzles I can help with [14:08:19] (03CR) 10Jean-Frédéric: "Looks good. See comments inline" (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280903 (https://phabricator.wikimedia.org/T131344) (owner: 10Lokal Profil) [14:12:53] (03CR) 10Jean-Frédéric: [C: 031] "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280890 (https://phabricator.wikimedia.org/T131483) (owner: 10Lokal Profil) [14:13:46] 6Labs, 10Tool-Labs, 10labs-sprint-116, 10labs-sprint-117, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#2168984 (10valhallasw) >>! In T113979#1681981, @valhallasw wrote: > There might be an issue with logging, though. We'd like to know who of the maintainers log... [14:17:39] (03CR) 10Jean-Frédéric: [C: 032] Updating Commons categories for se-bbr [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280890 (https://phabricator.wikimedia.org/T131483) (owner: 10Lokal Profil) [14:19:27] 6Labs, 10Tool-Labs: tools-bastion-01 is in SHUTOFF state - https://phabricator.wikimedia.org/T131135#2169023 (10chasemp) 5Open>3Invalid This was part of an ongoing investigation at the the time and will be deleted when possible. [14:20:43] (03Merged) 10jenkins-bot: Updating Commons categories for se-bbr [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280890 (https://phabricator.wikimedia.org/T131483) (owner: 10Lokal Profil) [14:21:15] 6Labs, 10Beta-Cluster-Infrastructure: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#2169029 (10Krenair) [14:25:34] (03PS1) 10Lokal Profil: import pywikibot instead of wikipedia [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280917 [14:31:21] 6Labs: VMs in labvirt1010 spontaneously enter SHUTOFF state - https://phabricator.wikimedia.org/T130802#2146936 (10Andrew) I think this happens when a virthost runs out of resources, either disk space or RAM. I've seen this happen during system reboots, if too many VMs are starting up at the same time. So, I s... [14:32:52] 6Labs, 10Phabricator: Login to phab-0[124].phabricator.eqiad.wmflabs is broken, even as root - https://phabricator.wikimedia.org/T130693#2169088 (10chasemp) p:5Triage>3Normal [14:33:46] 6Labs: VMs in labvirt1010 spontaneously enter SHUTOFF state - https://phabricator.wikimedia.org/T130802#2169099 (10Andrew) 5Open>3Resolved a:3Andrew So, action items here are: - Buy more virt nodes - During reboots, stagger 'nova start' commands further apart. I had this shaved down to 3 seconds per, bu... [14:52:34] (03PS1) 10Lokal Profil: Moving unusedImagesPage/imagesWithoutIdPage out of usernamespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280922 (https://phabricator.wikimedia.org/T131508) [14:52:48] 6Labs, 10Horizon: Proxy corner case: proxy name foo.wmflabs.org == domain name foo.wmflabs.org - https://phabricator.wikimedia.org/T131367#2169239 (10Andrew) I've just verified that there are no longer any current proxies bitten by this issue. It's still a UI issue though. [15:03:51] (03CR) 10Jean-Frédéric: [C: 032] Moving unusedImagesPage/imagesWithoutIdPage out of usernamespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280922 (https://phabricator.wikimedia.org/T131508) (owner: 10Lokal Profil) [15:05:06] (03CR) 10Jean-Frédéric: "Can you rebase André?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280922 (https://phabricator.wikimedia.org/T131508) (owner: 10Lokal Profil) [15:21:20] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2167379 (10Andrew) .19 is running now on https://labtestwikitech.wikimedia.org so y'all can see the problem. [15:23:53] as a reminder, on database replicas, please avoid long running queries with the following paterns "CREATE TABLE ... AS SELECT " "INSERT|UPDATE|DELETE ... SELECT " [15:24:27] you are delaying replication for every other user [15:33:09] 'Error: Unknown terminal type. [mosh is exiting.]' I get this wherever I try to use mosh. Any ideas why? My terminal is just set to use 'xterm' as a terminal type. [15:37:02] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2169368 (10bd808) >>! In T131445#2167674, @Parent5446 wrote: > Literally the only place that error message is used is in the AbortChangePassword hook... > > I think... [16:24:27] OK I’ve asked this question before, but I can’t for the life of me find the logs. Is there a captcha system available for a tool labs tool? Can I use a third party provider? Or will I have to write my own solution? [16:25:14] 6Labs: Measure capacity and utilization of labs services (Tracking) - https://phabricator.wikimedia.org/T107066#2169478 (10jcrespo) [16:25:16] 6Labs, 10DBA: Measure capacity and utilization of labsdb*** boxes - https://phabricator.wikimedia.org/T107070#2169474 (10jcrespo) 5Open>3Resolved Closing despite not having done anything publicly because it has been done privately on procurement at T131363. I will be happy to share the results of specs ASA... [16:39:39] Matthew_: No, I don't think so. I think that would require a disclaimer before loading it. Well… [16:40:04] 6Labs, 10Beta-Cluster-Infrastructure: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#2169522 (10greg) p:5Triage>3Unbreak! @Krenair: did you see anything note worthy when looking at this? [16:40:13] You could make a generic captcha based on mediawiki confirmedit extension (which provides wikipedia captchas) [16:40:42] actually, I had thought in making it standalone so it could be used by other kinds of webs [16:41:56] 6Labs, 10Beta-Cluster-Infrastructure: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#2169535 (10Krenair) When you try to start the instance, Horizon shows a success message in the UI. But when you go to the action log, you find the action listed with mes... [16:44:31] Platonides: That makes sense, I actually have an idea in mind. But I don't want to re-invent the wheel if it already exists. [16:47:26] ConfirmEdit is albe to create the images for a captcha [16:47:36] that's an independent python script [16:47:52] but you would need to reimplement the "asking-the-user", to not use mediawiki classes [16:48:01] that what should have a general alternative imho [16:48:37] Okay, that makes sense. Thank you. [16:49:38] There's a few captcha generators for PHP in Packagist (Composer) too, if you're using that. [16:50:48] tom29739: Do you have any specific recommendations? I can look into that as well. [17:02:18] 6Labs, 10Beta-Cluster-Infrastructure: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#2169622 (10Andrew) nova-compute says: 2016-04-01 17:01:24.328 56558 ERROR oslo_messaging.rpc.dispatcher [req-659dfa94-89f9-41cf-80a4-d81f1a42524a novaadmin deployment-p... [17:06:10] Matthew_, ironically, I was looking for one a couple of weeks ago. I'm using this one: https://github.com/gregwar/captcha at the moment. It seems to work fairly well, I can't show you any examples myself, but there's a picture of the output on that page. [17:06:13] 10PAWS: Write 2-3 year vision for annual plan - https://phabricator.wikimedia.org/T131267#2161651 (10Capt_Swing) +1 @yuvipanda. We can definitely make a case for PAWS supporting programming/data science education. And from there, we can even make the argument that by training new researchers using tools like PAW... [17:20:54] PROBLEM - Host tools-worker-1011 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:39] tom29739: Okay, thank you (Apologies, had to step away) [17:22:59] No problem. [17:36:23] hi, does anyone has problems to log into horizon.wikimedia.org ? [17:36:59] physikerwelt, it's requires 2FA to be enabled, do you have it enabled on your Wikitech account? [17:40:10] yes, I could log in the other day [17:40:29] !log telnet added cscott as user and project admin [17:40:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Telnet/SAL, Master [17:45:31] it keeps saying invalid credentials [17:47:00] physikerwelt: what 2fa do you use? [17:47:17] the google authenticator [17:47:22] on android [17:47:54] I'm having a similar issue atm [17:47:55] WFM [17:47:59] is you login to wikitech it works? [17:48:06] I was able to get on not long ago [17:48:07] for me it works [17:48:15] to log into wikitech [17:48:38] but not into horizon [17:50:26] but I could log in last week using the exactly same method [17:50:26] that I was using today [17:50:26] 6Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: watchlist table not available on labs - https://phabricator.wikimedia.org/T59617#2169753 (10Dispenser) @jcrespo Can we get any update? Its been some months and I have a few tools that can use this property. [17:54:03] 6Labs, 10Beta-Cluster-Infrastructure: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#2163472 (10thcipriani) hmm...would be nice to recreate that instance; however, the only role applied seems to be `role::beta::uploadservice` which just adds some text to... [17:56:16] 6Labs, 10labs-sprint-116, 10DBA, 13Patch-For-Review: watchlist table not available on labs - https://phabricator.wikimedia.org/T59617#2169800 (10jcrespo) Yes, although I have not updated it here, in order to correct several data issues and to populate the new labs server, all tables are being reloaded righ... [17:56:16] I was also able to change my password on https://wikitech.wikimedia.org/wiki/Special:Preferences which required 2fa [17:58:01] physikerwelt: can you log an issue and layout what you have tried [17:58:20] Reedy: do you use google authenticator by chance? [17:59:20] I fixed the problem, by signing up for 2fa again [17:59:39] ok [17:59:47] now I have a second number that is beeing generated that works [18:01:31] There's been a few problems with 2fa recently. Good thing that I don't need to use Horizon. [18:03:43] chasemp: yeah [18:04:06] Wikitech was broken earlier today partially by the .19 bump [18:07:42] I just successfully logged in to horizon [18:11:05] Krenair: do you have a moment to look w/ me why I can't get logged in? [18:12:13] chasemp, what's up? [18:14:01] chasemp: did you disable and enable 2fa in your account? that worked for me [18:14:08] I did [18:14:12] Krenair: pmd [18:14:18] I also changed my password [18:22:53] w.r.t. horizon I realized that I get logged out if I click on project e.g https://horizon.wikimedia.org/identity/math/detail/ [18:59:22] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170086 (10chasemp) [18:59:30] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170099 (10chasemp) p:5Triage>3Normal [19:06:11] 6Labs, 10Continuous-Integration-Infrastructure, 6Operations, 10Packaging: Update phantomjs to 2.1.1 on trusty - https://phabricator.wikimedia.org/T130940#2170134 (10Krinkle) 5Open>3declined There is no need for this. CI packages should be installed locally on a per-needed basis. This is just the need o... [19:07:50] 6Labs, 10Beta-Cluster-Infrastructure: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#2163472 (10hashar) deployment-upload is barely puppetized. It has been hacked in a few years ago as a way to emulate the media servers we have used before migrating to... [19:33:38] bd808: It worked ok using tools-dev, thanks for the help [19:53:00] (03PS2) 10Lokal Profil: Moving unusedImagesPage/imagesWithoutIdPage out of usernamespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280922 (https://phabricator.wikimedia.org/T131508) [19:58:52] (03PS4) 10Lokal Profil: Load project from config for unused images/images without id [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280903 (https://phabricator.wikimedia.org/T131344) [19:59:08] (03CR) 10Lokal Profil: Load project from config for unused images/images without id (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280903 (https://phabricator.wikimedia.org/T131344) (owner: 10Lokal Profil) [20:05:33] (03PS1) 10Lokal Profil: Change all linebreaks from Windows to Unix [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280969 [20:08:35] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2170288 (10csteipp) p:5Triage>3Unbreak! [20:15:55] 6Labs, 10Tool-Labs: Create replica.my.cnf in my home directory - https://phabricator.wikimedia.org/T131546#2170313 (10Wikitanvir) [20:46:11] !mosh [20:46:11] You should use mosh, you'll thank me later: https://en.wikipedia.org/wiki/Mosh_%28software%29 [20:47:34] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org, 13Patch-For-Review: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2170441 (10dpatrick) I have a patch in now. I'd like to reproduce the issue on https://labtestwikitech.wikimedia.org/, then have the fix deploye... [20:49:02] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org, 13Patch-For-Review: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2170442 (10Reedy) >>! In T131445#2170441, @dpatrick wrote: > I have a patch in now. I'd like to reproduce the issue on https://labtestwikitech.w... [20:54:02] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org, 13Patch-For-Review: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2170447 (10dpatrick) I have 2FA enabled (for use dpatrick2) and I get an error from that page. I might need those elevated permissions. [20:55:45] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org, 13Patch-For-Review: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2170449 (10bd808) >>! In T131445#2170447, @dpatrick wrote: > I have 2FA enabled (for use dpatrick2) and I get an error from that page. I might n... [20:56:01] 6Labs, 10Labs-Infrastructure: Sort out our libvirt qcow2 hack with the upstream - https://phabricator.wikimedia.org/T131548#2170450 (10Andrew) [21:04:16] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org, 13Patch-For-Review: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2170475 (10Andrew) @dpatrick, I just gave you 'shell' and 'cloudadmin' on labtestwikitech [21:04:30] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170476 (10chasemp) [21:06:39] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170477 (10chasemp) [21:10:21] 6Labs, 10Labs-Infrastructure: Sort out our libvirt qcow2 hack with the upstream - https://phabricator.wikimedia.org/T131548#2170492 (10Andrew) [21:10:23] 6Labs, 10Beta-Cluster-Infrastructure: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#2170491 (10Andrew) [21:11:49] (03PS1) 10Lokal Profil: Convert tabs to spaces and trim trailing whitespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281040 [21:13:52] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170498 (10tom29739) [21:14:06] 6Labs, 10Beta-Cluster-Infrastructure: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#2170499 (10Krenair) 5Open>3Resolved a:3Andrew [21:16:11] 6Labs, 7Tracking: DB rent apartment management for Wikimania 2016 Esino Lario - https://phabricator.wikimedia.org/T131551#2170517 (10Damiano64) [21:17:27] lol, phpmyadmin [21:18:57] Negative24: Are you currently here? [21:21:27] (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280922 (https://phabricator.wikimedia.org/T131508) (owner: 10Lokal Profil) [21:22:44] (03CR) 10Jean-Frédéric: [C: 032] Load project from config for unused images/images without id [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280903 (https://phabricator.wikimedia.org/T131344) (owner: 10Lokal Profil) [21:24:04] 6Labs, 10Beta-Cluster-Infrastructure: deployment-upload won't start, upload.beta.wmflabs.org down - https://phabricator.wikimedia.org/T131322#2163472 (10greg) >>! In T131322#2170491, @Andrew wrote: > YOUR job is to puppetize the damn thing :) Just for the record, that's mostly blocked on (afaiui): * {T64835}... [21:24:30] (03Merged) 10jenkins-bot: Load project from config for unused images/images without id [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280903 (https://phabricator.wikimedia.org/T131344) (owner: 10Lokal Profil) [21:25:44] (03CR) 10Jean-Frédéric: [C: 032] Moving unusedImagesPage/imagesWithoutIdPage out of usernamespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280922 (https://phabricator.wikimedia.org/T131508) (owner: 10Lokal Profil) [21:26:17] (03Merged) 10jenkins-bot: Moving unusedImagesPage/imagesWithoutIdPage out of usernamespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280922 (https://phabricator.wikimedia.org/T131508) (owner: 10Lokal Profil) [21:26:36] 6Labs: DB rent apartment management for Wikimania 2016 Esino Lario - https://phabricator.wikimedia.org/T131551#2170558 (10Peachey88) [21:30:06] (03PS1) 10Lokal Profil: Add requests to requirements [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281044 [21:34:09] (03CR) 10Jean-Frédéric: [C: 032] Add requests to requirements [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281044 (owner: 10Lokal Profil) [21:34:56] (03Merged) 10jenkins-bot: Add requests to requirements [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281044 (owner: 10Lokal Profil) [21:35:53] (03CR) 10Jean-Frédéric: [C: 032] Change all linebreaks from Windows to Unix [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280969 (owner: 10Lokal Profil) [21:36:37] (03CR) 10Jean-Frédéric: [C: 032] Convert tabs to spaces and trim trailing whitespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281040 (owner: 10Lokal Profil) [21:49:21] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170625 (10chasemp) > -- cgroups (trusty) -- this is primarily managed with the cgroup-bin package that includes cgroup-lite. cgroup-lite is a shim of an upstart service that (unconfigurable) manages to assoc... [21:50:19] 6Labs, 10Labs-Infrastructure, 10Tool-Labs: Backports are enabled in new Trusty instances - https://phabricator.wikimedia.org/T123890#2170626 (10Andrew) [21:50:24] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170628 (10chasemp) [21:50:49] 6Labs, 10Labs-Infrastructure, 10Tool-Labs: Backports are enabled in new Trusty instances - https://phabricator.wikimedia.org/T123890#1940220 (10Andrew) @MoritzMuehlenhoff, @csteipp, what is the policy on the backports repo for production servers? [21:53:37] (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280969 (owner: 10Lokal Profil) [21:53:55] (03CR) 10jenkins-bot: [V: 04-1] Change all linebreaks from Windows to Unix [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280969 (owner: 10Lokal Profil) [21:54:16] (03CR) 10Jean-Frédéric: "recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281040 (owner: 10Lokal Profil) [21:54:43] (03CR) 10jenkins-bot: [V: 04-1] Convert tabs to spaces and trim trailing whitespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281040 (owner: 10Lokal Profil) [21:54:48] 6Labs, 10Tool-Labs, 13Patch-For-Review: Install libbytes-random-secure-perl on tool labs - https://phabricator.wikimedia.org/T123824#2170640 (10Andrew) [21:54:51] 6Labs, 10Labs-Infrastructure, 10Tool-Labs: Backports are enabled in new Trusty instances - https://phabricator.wikimedia.org/T123890#2170638 (10Andrew) 5Open>3Resolved Labs team agrees that having this turned on is just fine. Moritz, pleaes re-open if this worries you. [22:01:59] 6Labs, 13Patch-For-Review, 5WMF-deploy-2016-04-05_(1.27.0-wmf.20): Add 'novaadmin' to new projects, always - https://phabricator.wikimedia.org/T131411#2170658 (10Andrew) p:5Triage>3Normal [22:02:32] 6Labs: Figure out what to do about servicegrouphomedirpattern - https://phabricator.wikimedia.org/T125002#2170659 (10Andrew) p:5Triage>3Normal [22:02:37] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170660 (10chasemp) > -- [[ http://linux.die.net/man/5/limits.conf | limits.conf ]] -- pam module that establishes resource limits and associations. So adding this to a module: > session required pam_li... [22:02:55] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170661 (10chasemp) [22:03:27] 6Labs, 10Labs-Infrastructure: Sort out our libvirt qcow2 hack with the upstream - https://phabricator.wikimedia.org/T131548#2170664 (10Andrew) p:5Triage>3Normal [22:03:47] 6Labs, 10wikitech.wikimedia.org: mariadb doesn't come up properly on silver after reboot - https://phabricator.wikimedia.org/T125987#2170665 (10Andrew) p:5Triage>3Normal [22:05:10] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 13Patch-For-Review, 5WMF-deploy-2016-04-05_(1.27.0-wmf.20): Clean up after ldap->mysql keystone migration - https://phabricator.wikimedia.org/T126758#2170667 (10Andrew) [22:08:00] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 13Patch-For-Review, 5WMF-deploy-2016-04-05_(1.27.0-wmf.20): Clean up after ldap->mysql keystone migration - https://phabricator.wikimedia.org/T126758#2170669 (10Andrew) [22:10:44] (03PS2) 10Lokal Profil: import pywikibot instead of wikipedia [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280917 [22:14:27] (03CR) 10Multichill: [C: 032] "About time :-)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280917 (owner: 10Lokal Profil) [22:14:35] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170762 (10chasemp) > -- [[ https://github.com/ghantoos/lshell | lshell ]] -- A pretty straight forward and simple "shell" wrapper that uses the cmd.cmd python module for interactive cli interfaces. I don't th... [22:14:50] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170765 (10chasemp) [22:15:27] (03Merged) 10jenkins-bot: import pywikibot instead of wikipedia [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280917 (owner: 10Lokal Profil) [22:19:28] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170773 (10chasemp) [22:23:15] 6Labs, 10Horizon, 13Patch-For-Review: Increase horizon session length - https://phabricator.wikimedia.org/T130621#2170774 (10Danny_B) [22:35:28] andrewbogott: Currently here? [22:40:11] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170836 (10chasemp) > tc -- userspace controls for the linux traffic QoS mechanisms, it is somewhat opaque and often complicated and the best documentation seems to be http://www.lartc.org/lartc.html#LARTC.QDI... [22:40:26] chasemp: Are you currently here? [22:40:29] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170838 (10chasemp) [22:40:34] yes [22:41:09] can you take a look at my instance? my ssh is broken after reboot. After the last labs reboot andrew need to start something manually [22:41:18] chasemp: It's rcm-2.rcm.eqiad.wmflabs [22:41:29] luke081515@bastion-01:~$ ssh rcm-2 [22:41:30] Permission denied (publickey,keyboard-interactive). [22:42:10] hm I can't get in [22:42:20] andrewbogott: can you hit that node? [22:44:09] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170849 (10chasemp) > iptables Limited bastions: * Because of the lack of ability with [[ https://phabricator.wikimedia.org/T131541#2170836 | tc ]] to do per user receive quota atm I am looking at using iptable... [22:44:32] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170850 (10chasemp) [22:46:27] 6Labs, 10Tool-Labs: Tools-bastion-05 is hanging, login is broken - https://phabricator.wikimedia.org/T131122#2170855 (10chasemp) [22:46:29] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170086 (10chasemp) [22:48:30] 6Labs, 10Tool-Labs: tools-bastion-05 is super slow - https://phabricator.wikimedia.org/T127992#2170860 (10chasemp) [22:48:32] 6Labs, 10Tool-Labs: overcommit disabled + small swap on tools-bastion-02 - https://phabricator.wikimedia.org/T123123#2170861 (10chasemp) [22:48:34] 6Labs, 10Tool-Labs: python3 triangle.py causing high memory usage on tools-bastion-02 - https://phabricator.wikimedia.org/T123122#2170862 (10chasemp) [22:48:36] 6Labs, 10Tool-Labs: Provide a tools-staging-bastion to allow users to test newly built packages - https://phabricator.wikimedia.org/T121146#2170863 (10chasemp) [22:48:38] 6Labs, 10Labs-Infrastructure: Set up ulimits on bastion - https://phabricator.wikimedia.org/T56719#2170864 (10chasemp) [22:48:40] 6Labs, 10Tool-Labs: Tools bastions are often unreliable - https://phabricator.wikimedia.org/T131541#2170865 (10chasemp) [22:51:02] !log tools.heritage JeanFred did a git pull for [[Phab:T131344]] and others [22:51:03] T131344: Change unused_monument_images.py to support other sites than Wikipedia - https://phabricator.wikimedia.org/T131344 [22:51:15] 6Labs, 10MediaWiki-extensions-OATHAuth, 10wikitech.wikimedia.org, 13Patch-For-Review: 2FA seems to be broken on wmf.19 - https://phabricator.wikimedia.org/T131445#2170869 (10dpatrick) Okay, I'm able to see the error now. Thanks! Is it possible, without too much trouble, to have https://gerrit.wikimedia.org... [22:51:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [22:59:53] (03PS2) 10Lokal Profil: Change all linebreaks from Windows to Unix [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280969 [23:00:27] (03PS2) 10Lokal Profil: Convert tabs to spaces and trim trailing whitespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281040 [23:04:14] 6Labs, 10Labs-Infrastructure, 10Tool-Labs: Backports are enabled in new Trusty instances - https://phabricator.wikimedia.org/T123890#2170882 (10csteipp) My only concern would be if were backporting security fixes there that were private (like we've done in the past with hhvm patches) [23:09:16] Hi [23:11:37] hi [23:17:35] 6Labs, 10Tool-Labs: Tool Labs users .bashrc file does not exist for tools accounts - https://phabricator.wikimedia.org/T131561#2170929 (10tom29739) [23:19:31] (03PS1) 10Lokal Profil: Mark Wikipedia tracking page edits as non-minor [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281053 (https://phabricator.wikimedia.org/T110830) [23:21:34] (03CR) 10Multichill: [C: 032] "\r\n" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281040 (owner: 10Lokal Profil) [23:23:00] (03CR) 10Multichill: [C: 032] "\s\r\n!" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280969 (owner: 10Lokal Profil) [23:23:33] (03Merged) 10jenkins-bot: Change all linebreaks from Windows to Unix [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/280969 (owner: 10Lokal Profil) [23:23:35] (03Merged) 10jenkins-bot: Convert tabs to spaces and trim trailing whitespace [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/281040 (owner: 10Lokal Profil) [23:40:29] !log mattermost killed the mattermost instance [23:40:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mattermost/SAL, Master [23:44:04] In the left menu, open the “Labs Projectadmins” section. [23:44:04] Click on “Manage Web Proxies”. [23:44:13] Where is it now? ^^^ :P [23:44:31] OuKB: horizon.wikimedia.org [23:44:44] \m/ [23:45:51] ergh, should I enable 2FA in wiki prefs or is it somewhere lese [23:47:44] OuKB, wasn't this explained on labs-l? [23:48:37] am I subscribed to it? :D [23:49:10] OuKB: yup, you need to enable it on wikitech [23:52:48] OuKB, you're expected to be [23:53:05] Krenair, I predate these expectations [23:56:51] 6Labs, 7Graphite: Setup "official labs grafana" instance - https://phabricator.wikimedia.org/T120295#2171030 (10yuvipanda) So I just found out that you can use stats from graphite.wmflabs.org on grafana.wmflabs.org - you can switch 'data sources' with the small icon at the bottom right of each graph config. So...