[00:01:28] PROBLEM - Puppet failure on tools-exec-1204 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:01:49] YuviPanda: I take it this is a recent manifest change and that you're already on it? [00:02:03] Coren: lots of discussion in -operations :) [00:02:58] PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:03:52] PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:09:02] RECOVERY - Puppet failure on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [00:12:56] 6Labs, 10Tool-Labs: Document how to turn shadow into master - https://phabricator.wikimedia.org/T91133#1812555 (10yuvipanda) 5Resolved>3Open Let's not close it until it does become true :) [00:18:47] RECOVERY - Puppet failure on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [00:19:13] RECOVERY - Puppet failure on tools-exec-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [00:20:33] RECOVERY - Puppet failure on tools-exec-1214 is OK: OK: Less than 1.00% above the threshold [0.0] [00:22:36] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [00:23:05] andrewbogott: ok the recoveries are starting now [00:23:14] andrewbogott: I'll remove role::labs::instance from LDAP after they're all done [00:23:16] RECOVERY - Puppet failure on tools-exec-1212 is OK: OK: Less than 1.00% above the threshold [0.0] [00:23:42] RECOVERY - Puppet failure on tools-exec-1216 is OK: OK: Less than 1.00% above the threshold [0.0] [00:23:54] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [00:24:04] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [00:24:08] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [00:24:42] RECOVERY - Puppet failure on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:25:26] RECOVERY - Puppet failure on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [00:26:52] RECOVERY - Puppet failure on tools-webgrid-generic-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [00:27:14] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [00:27:16] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [00:27:17] RECOVERY - Puppet failure on tools-exec-1220 is OK: OK: Less than 1.00% above the threshold [0.0] [00:27:48] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [00:27:59] RECOVERY - Puppet failure on tools-exec-1217 is OK: OK: Less than 1.00% above the threshold [0.0] [00:28:01] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [00:28:15] RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [00:28:27] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [00:29:13] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1201 is OK: OK: Less than 1.00% above the threshold [0.0] [00:29:27] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [00:30:25] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [00:31:05] RECOVERY - Puppet failure on tools-exec-1209 is OK: OK: Less than 1.00% above the threshold [0.0] [00:32:27] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [00:32:55] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [00:33:05] RECOVERY - Puppet failure on tools-exec-1203 is OK: OK: Less than 1.00% above the threshold [0.0] [00:33:27] PROBLEM - Puppet failure on tools-packages is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [00:33:55] RECOVERY - Puppet failure on tools-exec-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [00:34:12] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [00:34:14] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [00:34:16] RECOVERY - Puppet failure on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [00:34:24] RECOVERY - Puppet failure on tools-exec-1205 is OK: OK: Less than 1.00% above the threshold [0.0] [00:35:28] RECOVERY - Puppet failure on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [00:36:14] RECOVERY - Puppet failure on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [00:36:28] RECOVERY - Puppet failure on tools-exec-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [00:37:22] RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [00:37:22] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [00:37:23] RECOVERY - Puppet failure on tools-exec-1219 is OK: OK: Less than 1.00% above the threshold [0.0] [00:37:36] RECOVERY - Puppet failure on tools-exec-1218 is OK: OK: Less than 1.00% above the threshold [0.0] [00:37:58] RECOVERY - Puppet failure on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [00:38:10] RECOVERY - Puppet failure on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [00:38:11] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [00:38:11] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [00:38:52] RECOVERY - Puppet failure on tools-exec-1211 is OK: OK: Less than 1.00% above the threshold [0.0] [00:38:57] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [0.0] [00:40:21] Hi, all. I am ryuch using email address rcheol@gmail, a tool labs user. I lost my .my.cnf, How do I need to contact? [00:40:48] Cheol: can you file a bug on phabricator.wikimedia.org? then I can recreate it for your toool [00:42:44] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812599 (10Ryuch) 3NEW a:3yuvipanda [00:43:32] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812610 (10yuvipanda) What tool is this for? [00:44:17] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Spacemartin was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=204779 edit summary: [00:45:04] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812613 (10Ryuch) I am looking into Content Translation. So I am trying to install my MW instance. [00:45:07] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812614 (10yuvipanda) a:5yuvipanda>3None [00:45:43] looks like they only have access to one tool, YuviPanda [00:46:03] but that has a .my.cnf [00:46:10] as does their actual user [00:46:23] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812599 (10yuvipanda) I mean, did you create an account on tool labs and create a tool? If so, which one? Only tools get replica.my.cnf (and not .my.cnf)... [00:46:38] Krenair: can you put that on the ticket? [00:47:45] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812623 (10Ryuch) Yes. I did. My one is tools.ryu. [00:47:53] 6Labs, 10MediaWiki-extensions-OpenStackManager, 10Labs-Infrastructure: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1812624 (10yuvipanda) a:5yuvipanda>3None Oh, that's cool. I guess it still needs someone to manually delete the old host entries fr... [00:47:58] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812628 (10Krenair) ```krenair@tools-bastion-01:~$ ls -al ~ryuch/.my.cnf -rw------- 1 ryuch wikidev 50 Jul 28 2014 /home/ryuch/.my.cnf krenair@tools-bastion-01:~$ groups ryuch ryuch : wikidev project-bastion project-tools tools... [00:48:11] YuviPanda: I'm reminded of https://secure.phabricator.com/book/phabflavor/article/please_please_please/ [00:50:52] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812630 (10Ryuch) Dear Krenair, Those two are identical. It does not work for tools.ryu. [00:55:25] 6Labs, 10wikitech.wikimedia.org: Account creation fails - https://phabricator.wikimedia.org/T118916#1812655 (10Krenair) [00:57:39] 6Labs, 10wikitech.wikimedia.org: Account creation fails - https://phabricator.wikimedia.org/T118916#1812668 (10Krenair) ```2015-11-18 00:43:36 silver labswiki exception ERROR: [b5f59ffa] /w/index.php?title=Special:UserLogin&action=submitlogin&type=signup MWException from line 220 of /srv/mediawiki/php-1.27.0... [01:01:42] 6Labs, 10wikitech.wikimedia.org: Account creation fails - https://phabricator.wikimedia.org/T118916#1812686 (10Krenair) Also, I just successfully created an account. [01:02:44] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812687 (10Krenair) The files are identical, and the credentials in them do not work? [01:05:07] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812698 (10Ryuch) it's strange it works when I type just "my sql". Even I can create a database. But when I try to install my own MW, it fails because of the authentication. [01:08:14] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812705 (10Krenair) 5Open>3Invalid a:3Krenair [01:08:31] RECOVERY - Puppet failure on tools-packages is OK: OK: Less than 1.00% above the threshold [0.0] [01:14:48] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1812736 (10yuvipanda) a:3yuvipanda [01:18:22] 6Labs, 5Patch-For-Review: Convert all ldap globals into hiera variables instead - https://phabricator.wikimedia.org/T101447#1812753 (10yuvipanda) I've just removed all instances of ssh_hba from LDAP. [01:21:04] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812763 (10Ryuch) It is closed? But my problem remains. What shall I do? [01:23:48] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812764 (10Krenair) It sounds like the credentials work perfectly, but you haven't set MW up right. And I'm not sure you should be trying to use it in tools anyway... [01:24:09] Any ops around? [01:24:30] YuviPanda, ^ [01:24:38] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812765 (10yuvipanda) If you are trying to setup a mediawiki instance, I'll say tool labs is the wrong place for that a variety of reasons unless you are a seasoned Mediawiki admin. I'd suggest getting the language engineering t... [01:24:42] hi betacommand [01:26:59] Yuck can you kill all my scheduled tasks and run /bash/bots.sh ? [01:27:08] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812772 (10Ryuch) I have, tools.ryu@tools-bastion-02:~$ mysql --defaults-file="${HOME}"/.my.cnf ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) It works? [01:27:26] I'm currently experiencing a internet outage [01:27:36] betacommand: for which tool? [01:27:49] My only tool [01:27:52] betacommand: I can probably kill it all but wouldn't feel comfortable running a new batch. [01:28:12] YuviPanda: why not? [01:28:30] why I feel uncomfortable starting off someone else's bot that I haven't seen the code for? :) [01:29:09] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812778 (10Krenair) That should work if you set the database server hostname using `-h ` [01:29:19] Yuck it's just restarting the running tasks [01:29:39] *yuvipanda [01:29:43] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812779 (10yuvipanda) And that won't be the last problem you run into. I repeat my suggestion of getting added to the language engineering team's labs project and using Mediawiki-Vagrant in Labs. [01:30:10] betacommand: sorry, I still don't feel comfortable doing that? [01:30:30] if it blows up I'll have to debug with you over IRC and I'd rather not do that [01:30:43] debug with you as in relay things back and forth [01:31:11] YuviPanda, also, does sudo just work for you in tools w/o password? :P [01:31:15] Yuvi I can guarantee nothin g will blow up [01:31:27] MaxSem: yes. [01:31:36] heh [01:31:48] If something goes wrong feel free to kill them [01:32:03] betacommand, why you can't do it yourself? [01:32:34] But I'm not going to have ssh access for 24-36 hours due to a cut network line at my house [01:33:02] The outdoor cable broke [01:33:13] I'm posting from my phone atm [01:34:18] 6Labs, 10Tool-Labs: I lost my .my.cnf. - https://phabricator.wikimedia.org/T118915#1812800 (10Ryuch) Thanks, Kreanair and Yuvipanda. I take you advice, Yuvi. I agree it is closed. [01:34:37] betacommand: sorry, it's 5:30 PM here and I don't feel comfortable doing that with bots. plus restarting possibly stateful things from requests on IRC that I can't fully authenticate also make me uncomfortable, so... [01:34:56] 6Labs, 10wikitech.wikimedia.org: Account creation success but shows error page - https://phabricator.wikimedia.org/T118916#1812801 (10Niedzielski) [01:35:15] betacommand: are they just the IRC bots? [01:35:31] Earwig: yeah [01:35:49] YuviPanda: BC's IRC bot has been down in the bot approvals channel for a while now and it's been annoying :/ [01:36:06] hmm this is all kinds of complicated now [01:36:19] with webservices they're stateless so I can just bounce them [01:36:27] YuviPanda: would a phab ticket be enough? [01:36:44] I'm going to say no and suggest that the right thing to do is for betacommand to give other people access and run each bot as a separate tool account so that there can be granular access [01:36:52] 6Labs, 10wikitech.wikimedia.org: Account creation success but shows error page - https://phabricator.wikimedia.org/T118916#1812824 (10Krenair) (without getting an error page) [01:37:12] YuviPanda: the irc bots are stateless [01:37:13] betacommand: I can probably add someone else to your tool if you want to with a phab ticket or email verification. [01:37:37] Rather not atm [01:37:51] well, not much I can do. I'm not a substitute for having multiple maintainers, sorry. [01:41:14] Wow.... Shocked that the lack of basic support. The one of the few times I ask for something really basic I'm given a cold shoulder [01:41:22] well [01:42:44] I don't really think I can help with the individual tools lacking multiple maintainers (this exact scenario is why we suggest people run each tool individually and share out access) [01:43:27] Yuvi all I am asking for is to restart my tasks [01:44:08] sorry betacommand. if I were actually familiar with your tools I'll be far more inclined to do that, but I'm not and I also have to leave in about 15m somewhere else and I'd rather not do that. [01:44:24] I guess I'll try to do it thru my phone since I'm getting no help from the labs roots [01:44:32] sure, it's all my fault [01:45:29] lol [01:45:55] Yuvi the code has been running since 2006 with little code change. On occasion they dc from IRC and need restarted. [01:45:56] betacommand: dunno what kind of phone you have, but I have an app that allows me to SSH to computers for urgent situations like this [04:27:06] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1812974 (10Ricordisamoa) I'm going **bold** and pinging @Magnus who might be interested. [05:13:09] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813000 (10Harej) Do we want moving some of his tools into production as a goal to aspire to? [05:19:32] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813005 (10NiharikaKohli) >>! In T115537#1812974, @Ricordisamoa wrote: > I'm going **bold** and pinging @Magnus who might be interested. Hi @ricordisamoa, if this is a project... [05:29:58] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813018 (10Ricordisamoa) >>! In T115537#1813000, @Harej wrote: > Do we want moving some of his tools into production as a goal to aspire to? Some of them might benefit from pro... [06:00:28] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813057 (10Ricordisamoa) >>! In T115537#1813005, @NiharikaKohli wrote: > Hi @ricordisamoa, if this is a project you'd like the Community Tech team to undertake, I suggest you ad... [06:36:19] 10Wikibugs, 6Phabricator: Wrong message, wikibugs displayed normaly users, if herald does an action - https://phabricator.wikimedia.org/T116477#1813105 (10Luke081515) [06:49:05] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [07:12:59] if I have access to a given wikitech project, I should be able to log in to any node in the project, right? [07:29:00] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [08:28:21] 6Labs, 6operations: Can not access new public IP despite security settings - https://phabricator.wikimedia.org/T118936#1813172 (10yuvipanda) 3NEW [08:29:05] 6Labs, 6operations: Can not access new public IP despite security settings - https://phabricator.wikimedia.org/T118936#1813179 (10yuvipanda) So port 8065 is also open via a security group to just inside labs, and I can telnet that. So I changed port 6667 to be open to just inside labs and *can not telnet there... [08:31:29] 6Labs, 6operations: Can not access new public IP despite security settings - https://phabricator.wikimedia.org/T118936#1813181 (10yuvipanda) [08:49:49] 6Labs, 6operations: Can not access new public IP despite security settings - https://phabricator.wikimedia.org/T118936#1813206 (10yuvipanda) I've tried a bunch more things: - Moving it to port 9000, and trying 10.0.0.0/8 security group (no luck!) - Moving it to port 9000, and trying 0.0.0.0/0 security group (... [09:21:24] 6Labs, 6operations: Can not access new public IP despite security settings - https://phabricator.wikimedia.org/T118936#1813221 (10yuvipanda) With some debugging help from @akosiaris, it turns out that new security rules aren't being applied on labvirt1010 until a nova-compute restart (I had to restart twice) [09:32:01] 10Wikibugs, 6Phabricator: Case of wikibugs displaying unrelated user when Herald performed an action - https://phabricator.wikimedia.org/T116477#1813232 (10Aklapper) p:5Triage>3Lowest [09:53:29] YuviPanda: we should probably add a parameter to require_package to allow 'latest' instead of 'present' [09:53:53] valhallasw`cloud: right. we can also partially revert that change since just getting rid of python-yaml would've also been enough [09:54:16] that would also get rid of python3-yaml :P [09:54:25] valhallasw`cloud: I'm not sure if puppet allows named parameters though, so if it doesn't idk how we can both support arbitrary number of packages and latest flag [09:54:28] that's true [09:55:39] I'm also confused, I thought it /did/ ensure=>latest [09:55:47] because https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/lib/puppet/parser/functions/ensure_packages.rb already allows the ensure=>present use case [09:56:05] oh, but that has to do with ordering [09:56:22] valhallasw`cloud: ensure_package takes a list as first param [09:56:27] and require_packages doesn't [09:56:33] this is I suppose a bug in require_packages [09:56:55] require_packages /can/ take a list [09:58:05] valhallasw`cloud: right, but since it calls args.flatten it can't take anything more [09:58:26] so I guess fix is to force the first param to be a list or a string, and then we can have second params [09:58:53] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813261 (10Magnus) Finally, someone realized just how much shoddy code I deployed! :-) Some thoughts on the individual points: - Standardize deployment: I do try to apply pull... [09:58:57] mrm. [10:00:51] YuviPanda: alternatively... we can just do an aptitude dist-upgrade after/before a puppet run? [10:01:11] doesn't that also make *everything* latest [10:01:16] not just the stuff we have? [10:01:24] yes [10:01:31] which I think we probably want anyway? [10:01:42] actually [10:01:51] we already theoretically have unattended upgrades set [10:02:01] maybe that's good enough? [10:02:40] mm. Not sure, try sudo apt-get -u upgrade on a host [10:03:13] but maybe those are just a remainder from the unattended upgrades [10:03:32] also it should be connected to the puppet run if possible [10:04:37] 0 Unattended-Upgrade::Allowed-Origins { [10:04:40] 1 "${distro_id}:${distro_codename}-security"; [10:04:46] valhallasw`cloud: we're only doing security updates [10:07:15] valhallasw`cloud: let me file a bug [10:07:22] already doing [10:07:35] valhallasw`cloud: <3 [10:07:43] valhallasw`cloud: want to do the swap thing now? [10:07:51] I Fucked up my sleep cycle might as well do something useful [10:07:58] no, at work :-p [10:08:09] ah ofc [10:08:22] valhallasw`cloud: do you mind if I do it tomorrow even if you aren't around? [10:08:33] 6Labs, 10Tool-Labs: Make sure all packages are automatically upgraded - https://phabricator.wikimedia.org/T118939#1813269 (10valhallasw) 3NEW [10:09:11] no, that's fine. You can also do it now, I just can't babysit the deploy atm [10:11:23] valhallasw`cloud: yeah sure. [10:11:27] nah I should do it tomorrow [10:11:32] and try to unfuck sleep cycle [10:15:31] sounds sane [10:15:34] good night :-) [10:16:14] I'll try :) [10:16:28] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813293 (10TheDJ) > As for "productizing" some of my tools I think when we are talking about that, it is essential to not only look at an 'end product', but also to see if we c... [10:29:49] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813314 (10Magnus) There are certainly some "code blocks" that could be abstracted; however, I think we should be careful not to create Yet Another Wikipedia/Wikidata PHP librar... [11:28:04] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1813406 (10Joe) FTR. this just happened to me with a newly-created instance with jessie; to my knowledge no prior machine with that name existed and puppet is failing even after a reboot: ``... [11:28:14] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1813407 (10Joe) p:5Normal>3High [11:33:55] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813414 (10Harej) >>! In T115537#1813314, @Magnus wrote: > Another approach to "building blocks" are lists of pages/items that can be exchanged between tools. I have made severa... [12:55:36] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813509 (10TheDJ) PagePile is exactly the type of the element that I was thinking about in my comments. [13:14:44] !log rcm deleted rcm-3, tests done, so currently unused [13:14:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL, Master [13:25:26] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813544 (10Ricordisamoa) Of course I bow to Magnus for his astounding contributions to the movement. My words shall by no means be intended as criticism of him or his deeds. On... [13:35:32] 10Tool-Labs-tools-Other, 6Community-Tech, 7Tracking: Improving Magnus' tools (tracking) - https://phabricator.wikimedia.org/T115537#1813553 (10Magnus) No offense was taken :-) I am very well aware that my code is not exactly up to industry standard. Anything to improve that, within my time bandwidth, is wel... [14:17:53] Hello! Anyone else unable to login to wikitech? I'm getting internal error. [14:19:14] andrewbogott: ^ [14:19:32] Niharika, cannot reproduce [14:19:50] can you try again, there was a recent issue, Niharika [14:20:24] jynus: Ah, trying a couple of times worked. Thanks! [14:20:37] sorry, about that, Niharika [14:20:45] but happy it finally worked [14:21:01] That's alright. :) [14:27:23] Niharika: looking... [14:27:38] andrewbogott: I got it sorted, no worries. [14:27:56] ok! [14:28:04] jynus: ‘recent issue’? [14:28:15] andrewbogott, see -ops [14:28:24] not labs-specific [14:31:00] 6Labs, 3Community-Tech-Sprint: Labs project for the Community Tech team - https://phabricator.wikimedia.org/T118944#1813600 (10NiharikaKohli) 3NEW a:3NiharikaKohli [14:32:00] 6Labs, 3Community-Tech-Sprint: Labs project for the Community Tech team - https://phabricator.wikimedia.org/T118944#1813600 (10NiharikaKohli) [15:31:47] anybody know where the code for tools.wmflabs.org/guc lives? wanting to fix T118662 [15:34:32] douglas: https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/guc [15:35:42] Glaisher: thanks :) [15:45:02] deploy sync is broken on deploylabs for graphoid (( [15:45:14] YuviPanda, ? [15:45:34] i keep forgetting who knows about the trebuchet in labs [15:51:36] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1813785 (10Andrew) It looks like you deleted that instance... is that right? If so, can you see if you're able to repeat the issue and ping me with the failed instance? [15:53:00] 6Labs, 3Community-Tech-Sprint: Labs project for the Community Tech team - https://phabricator.wikimedia.org/T118944#1813796 (10Andrew) 5Open>3Resolved OK, created! User Niharika29 can add additional members and admins as you see fit. [15:53:45] 6Labs, 3Community-Tech-Sprint: Labs project for the Community Tech team - https://phabricator.wikimedia.org/T118944#1813805 (10NiharikaKohli) Thanks! [16:06:40] (03PS1) 10Alexandros Kosiaris: Add various secrets [labs/private] - 10https://gerrit.wikimedia.org/r/253918 [16:10:22] Hi, I created a new project and instance. I'm unable to SSH to the provided DNS name. Is there an additional step I'm missing? [16:12:29] I recall there was a handy guide to this by bd808, but I can't find it. [16:12:34] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Add various secrets [labs/private] - 10https://gerrit.wikimedia.org/r/253918 (owner: 10Alexandros Kosiaris) [16:13:01] Niharika, maybe this? https://wikitech.wikimedia.org/wiki/Help:Access [16:18:18] jynus: Thanks, that was helpful! [16:38:25] One more question. I'm trying to vagrant up in my instance, but it says: "No usable default provider could be found for your system.". What could possibly be causing this? [16:39:26] Niharika: All the good docs are at https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs [16:39:58] bd808: Yes, but no troubleshooting section. :( [16:41:03] Niharika: the "No usable default provider could be found for your system." error sounds like the vagrant alias is missing in your shell environment [16:41:28] Niharika: "5. Log out and log back in to pick up profile.d alias that will make the vagrant command run Vagrant as the mwvagrant shared user account." [16:41:58] bd808: I saw that but "pick up profile.d alias" was confusing. I also don't see that anywhere. [16:42:05] I did logout and in again. [16:42:40] `alias|grep vagrant` should say something like "alias vagrant='/usr/local/bin/mwvagrant'" [16:43:30] bd808: Yes, it says that. [16:43:44] hmm... [16:44:15] And yet complains about no usable provider when I do a vagrant up. [16:48:52] if I understand the vagrant docs correctly, the 'provider' is the software that provides the virtual machine (virtualbox, vmware, ...) [16:49:11] yeah. it should be seeing that lxc is available [16:49:13] valhallasw`cloud: Right. [16:49:25] Niharika: can I poke around on the instance and see if I can figure it out? [16:49:37] bd808: Sure, lemme give you access. [16:51:00] bd808: Done. It's commtech-1.commtech.eqiad.wmflabs [16:53:39] !log commtech Fixed permissions on commtech-1 with `sudo chown mwvagrant /srv/mediawiki-vagrant/.vagrant` [16:53:53] Niharika: ^ I'm trying to bring up the vm now [16:54:17] Awesome. [16:55:07] Think bd808 could've granted himself access, Niharika ... [16:55:44] Krenair: Really? I thought only Labs admins could do that magic. [16:55:55] * bd808 has super powers in labs [16:56:04] He is. bd808 is a labs cloudadmin, Niharika [16:56:08] Niharika: here's what I did -- https://phabricator.wikimedia.org/P2328 [16:56:14] Oops. My bad. :) [16:56:35] No worries. I asked nicely because I didn't want to just barge in on your toys [16:57:10] * bd808 goes to a meeting [16:57:18] bd808: Thanks a ton! [16:57:46] Niharika: the initial vm is still provisioning. I'll ping you when it is done [16:57:58] Got it. [17:06:37] Niharika: looks like it is up and running now [17:15:34] bd808: Great, thank! [18:22:11] 6Labs, 7Database: Database replicas: replicate user.user_touched - https://phabricator.wikimedia.org/T92841#1814320 (10jcrespo) This requires some planning: drop and update the triggers, which has to be done while replication is stopped to avoid leaks. I've started to do so at: https://gerrit.wikimedia.org/r/2... [18:22:30] Coren, do you have a few minutes? [18:22:39] Or a minute for that matter. [18:22:46] Cyberpower678: What's up? [18:23:07] What's the password to login to fabricator? [18:23:12] *phab [18:23:44] Cyberpower678: LDAP, or if you login via the mediawiki button, SUL [18:24:10] LDAP = wikitech [18:24:13] I used LDAP, but ever since I've changed my passwords on Wikitech, I seem to be locked out. [18:24:42] When I type in my Wikitech password, it comes back invalid, but works fine on Wikitech. :/ [18:24:57] Could it have something to do with 2fa? [18:25:19] Cyberpower678: Odd; I've heard no reports of issue in that direction. I have 2fa on and it works fine on my end. Lemme test now. [18:25:26] Cyberpower678: can you login to gerrit? [18:25:35] also, try your old password? [18:25:41] I've never ever tried to login to gerri. [18:26:37] Cyberpower678: wfm. How confident are you that you have the right password? :-) (Also, remember username is case sensitive and uses your wikitech username not shell account) [18:27:47] There we go. [18:27:55] It was the username. [19:32:01] 6Labs, 10wikitech.wikimedia.org, 7Security-General: Add password requirements for wikitech accounts - https://phabricator.wikimedia.org/T118751#1814534 (10Ricordisamoa) [19:46:37] PROBLEM - Host tools-andrew-puppettest is DOWN: CRITICAL - Host Unreachable (10.68.21.109) [19:48:27] andrewbogott: ok I'm going to kill ^ now [19:48:39] YuviPanda: thank you :) [19:48:51] I've no idea why it happens [20:04:40] andrewbogott: shinkengen didn't like the lack of any roles (hah!). I've fixed that now [20:30:49] 6Labs, 10wikitech.wikimedia.org: Account creation success but shows error page - https://phabricator.wikimedia.org/T118916#1814821 (10Aklapper) Same problem here: Error but successfully logged in. Maybe a side effect of {T118751}? [20:48:00] hi. What is the purpose of the RSA key when requesting in Special:OAuthConsumerRegistration, "Please provide a public RSA key (in PEM format) if possible"? [20:48:31] Mpaa-irc: I think csteipp or anomie would know [20:49:23] Mpaa-irc: it allows you to provide a public key to the server rather than the server providing a private key to you. I'm not entirely sure why that would be useful in an OAuth context, though; I never filled in that field, and that works without issues. [20:50:11] valhallasw`cloud, YuviPanda OK, thanks [20:50:35] valhallasw`cloud: I've setup a 4pm slot to do the swap space stuff. [20:50:56] Coren: did you ever get to valhallasw`cloud's question about if there was a specific reason the webgrid nodes had no swap? [20:50:58] then I will happily be asleep [20:50:59] I hope [20:51:06] valhallasw`cloud: better be! [20:53:24] YuviPanda: No. I missed the question I think. [20:53:54] Coren: ok. the exec nodes had a large amoutn of SWAP (2x RAM?) while the web nodes did not. was that intentional? [20:55:22] YuviPanda: It was, but that might not be an issue anymore. Web nodes used to allow overcommit (unlike normal exec nodes) [20:56:25] Coren: in SGE, yes, but the kernel did (and does) not. [20:57:09] Hm. [20:57:31] It did in a previous epoch; but you're correct that overcommit is now off on any exec_environ node. [20:57:41] So they should have swap too. [20:57:53] toollabs::node::compute::general sets the swap [20:58:00] Mpaa-irc, valhallasw`cloud: It allows you to use oauth_signature_method=RSA-SHA1 instead of oauth_signature_method=HMAC-SHA1. [20:59:04] webgrid hosts are not compute hosts [20:59:27] I know. I'm saying they *should* have swap, not that they currently do. :-) [20:59:55] Ah, right. Yes, this is what Yuvi and I figured out last week. There's a patch in Gerrit to add swap to all exec hosts [21:00:05] anomie, thanks [21:00:10] (also the dedicated ones) [21:00:19] move from toollabs::node::compute::general to exec_environ? [21:00:30] to a new class toollabs::node [21:00:59] Ah, no. I see the inheritance bit. Better way indeed. [21:01:08] err [21:01:10] inheritance sucs [21:01:12] *sucks [21:01:14] we should use include [21:01:20] but now that I think of it, it might be better to also move the sysctl::parameters there [21:01:21] it makes any dealing with hiera super complex [21:01:48] YuviPanda: Ah, didn't consider hiera. That was otherwise a clearcut good use case for inheritance. :-) [21:01:51] so that exec_environ only handles software, while node handles the more infra side of things. Oh well. [21:02:13] yes, include is probably better [21:02:15] valhallasw`cloud: No, it's still okay; just include the ::node class rather than inherit from it. [21:02:50] yeah, it's currently inheritance because webgrid hosts used inheritance, but they probably don't need to [21:02:59] err, node::web [21:03:29] yeah, the entire thing needs to be cleaned up from inheritance at some point [21:03:39] ori keeps shaming me about it now and then :) [21:04:30] YuviPanda: http://wdq.wmflabs.org/stats is stalled [21:04:33] Any idea why? [21:04:44] hmm no [21:04:49] I can restart it maybe :) [21:04:55] 2015-11-17T08:39:30Z is quite some time ago [21:05:01] which is the only thing I can do to that really since the C++ is above my head [21:05:01] Can you see if both nodes report the same? [21:05:58] YuviPanda: I think the inheritance can safely go. Let me go do that. [21:06:09] valhallasw`cloud: \o/ [21:06:24] Coren: do you think you can babysit and merge that patch *today*? if you're busy with other things no problem, I can do that [21:06:28] I'm confused why toollabs::node::web includes toollabs and toollabs::node::genreal does not? [21:06:30] but i'd want to get this done with today [21:06:40] valhallasw`cloud: yeah, that entire thing is inconsistent and confusing [21:06:52] probably a role thing [21:07:00] YuviPanda: Can you see if both nodes report the same? [21:07:12] multichill: yup am sshing now to find out [21:07:19] YuviPanda: Sure. valhallasw`cloud: wanna switch to includes or should I? [21:07:46] Coren: I think he's doing that now. [21:07:48] Coren: if you could do that, that would be awesome -- I'm actually sort of heading to bed [21:07:50] ah [21:07:52] :D [21:07:54] ok [21:08:05] I was still in the prep stage 'read puppet manifests and try to figure out what the current entangled logic is' [21:08:06] valhallasw`cloud: No worries. Goto sleep. [21:08:08] Coren: this also needs a restart on precise I think - do verify that all nodes have the swap setup afterwards [21:08:36] YuviPanda: It shouldn't, really - all it needs is a swapon -a [21:09:07] Coren: sure, but it needed it when I was setting it up last time - I think the /tmp needs a reboot, not the swap [21:09:23] Coren: either way, make sure that they both work and verify, etc :) [21:09:29] Ah, yes, /tmp definitely does. [21:09:44] webnodes we can restart more easily [21:09:46] so [21:09:50] I would have expected the bigger /tmp could wait until next restart for other reason though - nothing relies on it being there [21:10:00] Coren: ok, just checked. toollabs is already included via toollabs <- toollabs::compute <- role::labs::tools::compute <- webnode, so that inheritance in node::web can safely be removed completely [21:10:27] Coren: I don't think we should have ticking 'bombs' that change behavior at random points in the future [21:11:05] I don't mind either way; but I'll wait until I return from the gym to do the merge then. [21:11:44] Coren: ok [21:11:53] Coren: and do !log as you do things :) [21:11:57] * YuviPanda goes to go to the office [21:20:36] Hi there. How do I properly escape spaces in the arguments to my program in jsub? [21:21:00] when I run it locally, it works: myprog "arg arg" and myprog arg\ arg [21:21:29] but when I run this via jsub, I receive two separate args instead of one with a space in between [21:21:33] what do I do? [21:21:40] YuviPanda: Found anything? [21:21:54] multichill: fighting with a stupid ssh config >_> [21:21:58] gimme a few mins [21:22:14] You mentioned something about escaping to the office [21:22:27] Leloiandudu: You'd have to double quote; but that's unreliable in the best of cases. I would very much recommend that you have a shell script with your invokation and jsub /that/ instead. [21:23:01] multichill: yeah too many things to do. currnetly still on couch [21:23:37] I know the feeling. Got in on time and left around 19:00 so I had a pretty long day too [21:27:29] Coren: what do you mean by double-quoting? the whole string after jsub? [21:28:20] Leloiandudu: Yes, you could do "'arg arg'" but I very much recommend against it. Passing arguments into shells that require escaping is always iffy in the best of cases. [21:35:08] Coren: thanks for you help! ^^ [21:36:01] multichill: I see a bunch of [21:36:05] > MySQL server has gone away [21:36:08] on logs [21:36:10] no timestamps :( [21:36:11] I restarted [21:36:46] multichill: and it's slowly reading allll the things from some file somehwere [21:36:59] * YuviPanda gives it time, gets dressed to go to office [21:55:31] YuviPanda: Seems to be catching up [21:55:42] multichill: \o/ [21:55:47] (I'm still on the couch dammit) [21:56:10] YuviPanda: Did you mess up and also send admin password reminders for locked down lists? :P [21:56:26] multichill: I've no idea. you should ask mutante [21:56:42] I don't even know we have locked down lists or what they mean [21:57:09] pywikipedia-svn and pywikipedia-l should be closed