[06:50:22] 06Labs: cronspam from labscontrol1001 and labstore1001 - https://phabricator.wikimedia.org/T132422#2198026 (10elukey) [06:50:57] 06Labs: cronspam from labscontrol1001 and labstore1001 - https://phabricator.wikimedia.org/T132422#2198041 (10elukey) [06:57:47] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2198047 (10elukey) [07:13:25] 06Labs, 10Tool-Labs, 10Tool-Labs-tools-Other, 10DBA: Throttling Cyberbot tool user as it is consuming most of the CPU - https://phabricator.wikimedia.org/T131937#2198074 (10Volans) @Cyberpower678 apart some general advice at the end it really comes to the queries that the application will perform, how the... [07:53:15] (03PS78) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [07:58:58] (03CR) 10Ricordisamoa: "PS78 passes data in the content of a new script#slicer-data element instead of as data-slicer attribute in div#game" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [08:03:31] (03CR) 10Ricordisamoa: "Example response length on real data:" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [08:07:26] (03CR) 10Ricordisamoa: "PS74: 1.3M" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [08:34:51] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:46:53] 06Labs, 10Labs-Infrastructure: I/O on labmon1001 is very slow - https://phabricator.wikimedia.org/T127957#2198152 (10fgiunchedi) yeah, without getting a new machine we could add the two unused disks to a degraded raid10, rsync 825G onto it, and add the two disks from md127 to the raid10. that might or might no... [10:18:02] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2198320 (10Gunnex) So far I can analyze the situation (I am not a technician, just giving some feedback from the user front about what I am currently monitoring...), only... [10:19:06] RECOVERY - Puppet run on tools-flannel-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:11:19] hmm [12:11:33] i can't seem to add puppet roles from wikitech anymore and i don't see a way to do it in horizon yet [12:17:44] RECOVERY - Puppet run on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [12:19:25] 06Labs, 07Documentation: Can't figure out how to add puppet roles to a labs instance anymore (gone from wikitech, not found on horizon) - https://phabricator.wikimedia.org/T132437#2198536 (10brion) [12:19:28] filed as https://phabricator.wikimedia.org/T132437 [12:20:41] it's still showing all instances under "Manage instances" for me [12:23:22] 06Labs, 07Documentation: Can't figure out how to add puppet roles to a labs instance anymore (gone from wikitech, not found on horizon) - https://phabricator.wikimedia.org/T132437#2198550 (10brion) Instance list on wikitech showing no instances for ogvjs-integration project: {F3865388} Instance list on horizo... [12:23:32] moritzm: i've got none [12:23:34] :( [12:25:17] brion: I don't have further details, just wanted to provide a data point that it's not universally broken. Andrew should be online soon [12:25:30] thx [12:29:50] *yawn* well it's past bedtime for me ;) i'll check in in a while [13:04:28] Did something change on Tool Labs? [13:04:33] https://www.irccloud.com/pastebin/N4CzQOrz/ [13:05:36] yes, fingerprint changed, unsure if the new are on the wiki [13:07:50] where can i find this new fingerprint? [13:07:58] or, i guess i can just delete it and it will create a new one for me [13:09:40] problem solved. thanks! [13:17:45] Hrm. I'm having issues authenticating against horizon; "an error occured authenticating." andrewbogott, chasemp? [13:18:51] ... may be related to the fact that wikitech itself also seems ill. [13:30:41] (03PS79) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [13:31:20] Am I the only one having issues w/ logging into horizon? [13:38:39] (03CR) 10Ricordisamoa: "PS79 makes it work for sitelinks" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [13:39:33] chasemp: yt? [13:43:36] Not really as I am packing to fly out :) andrew will be around here any minute [13:49:27] Where to? [14:02:28] paradise [14:03:03] o_O? [14:23:30] Hi, I'm having trouble on Labs. I want to update composer dependencies, but it says 'composer':command not found. Same for 'php'. I've a mediawiki-vagrant instance on labs. [14:30:47] Duh, ignore me. I didn't ssh to vagrant. [14:35:17] lol [14:45:47] (03PS80) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [14:49:36] (03CR) 10Ricordisamoa: "PS80 makes each Section class control its own validity rules" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [15:02:46] (03PS81) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [15:09:03] (03CR) 10Ricordisamoa: "PS81 moves getValidityRules() into EditableDraggableElement's prototype" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [15:12:09] (03PS82) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [15:14:37] (03CR) 10Ricordisamoa: "PS82 makes the pipe character in labels and descriptions valid again" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [15:20:14] 06Labs, 07Documentation: Can't figure out how to add puppet roles to a labs instance anymore (gone from wikitech, not found on horizon) - https://phabricator.wikimedia.org/T132437#2198536 (10yuvipanda) Have you tried logging off and back on to wikitech? I bet 20$ that'll fix the issue [15:27:08] andrewbogott: the instance commtech-1.commtech.eqiad.wmflabs in the commtech project is being a pain. Neither horizon nor wikitech will give me links to see its console or reboot. [15:27:39] when did it last work [15:27:40] ? [15:27:54] Niharika was working on it a few hours ago [15:27:57] andrewbogott: An hour ago or so. [15:28:16] andrewbogott: About an hour or so ago, I sshed in successfully, then fetched Jon Robson's patch on a new branch, then ssh-ed to the VM and did 'composer update'. It updated a lot of things but got stuck on 'sebastian/exporter' and eventually my connection timed out. Unable to ssh in since then. [15:28:50] Horizon says 'Error: Unable to retrieve details for instance "f71161fd-25e1-4656-b697-9f3a2c8b88cd".' wikitech shows it in https://wikitech.wikimedia.org/wiki/Special:NovaInstance with nothing in the "Actions" column [15:29:00] looks like you tried a reboot? [15:29:23] | f71161fd-25e1-4656-b697-9f3a2c8b88cd | commtech-1 | ACTIVE | - | Running | public=10.68.18.236 | [15:29:30] I was going to, but no UI to do so [15:30:34] All the webui stuff is working for me (horizon and wikitech both) [15:30:44] the syslog looks pretty unhappy though [15:30:50] I'll reboot it [15:31:00] thanks [15:31:16] !log commtech rebooting instance commtech-1 [15:31:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Commtech/SAL, dummy [15:32:13] andrewbogott: Awesome, thanks. I can ssh in. [15:32:17] sure [15:32:36] I don't know why it died. [15:32:50] bd808: are you a long-term member of the commtech project or did you just add yourself? [15:32:55] andrewbogott: Should I re-attempt a 'composer update'? [15:33:08] andrewbogott: bd808 has been a member for quite a while. [15:33:18] andrewbogott: I think I created the project [15:33:33] Niharika: I can't really offer any advice about what to do with the instance. [15:33:42] andrewbogott: Okay, thanks anyway. [15:33:50] I've never used composer [15:33:53] oh... I'm not showing as an admin for it [15:33:56] huh [15:34:02] bd808: I created it. [15:34:08] bd808: But you helped. [15:34:44] andrewbogott: mystery solved for the horizon and wikitech links [15:34:53] bd808: ok! [15:34:56] I was a project member but not a project admin [15:38:15] bd808: ok, so that is maybe a subcase of https://phabricator.wikimedia.org/T132187 [15:38:37] feel free to comment on that task, it's most likely going to consist of a million little fixes [15:38:47] "fun" [15:39:32] Horizon should always say at the top "Your roles in this project are: " [15:39:45] But we're basically the only deployment that uses more than one role [15:40:00] so we keep falling into these "we didn't think of that" pits [15:48:30] 06Labs, 10Horizon: Horizon behavior for non-projectadmins is weird - https://phabricator.wikimedia.org/T132187#2190880 (10bd808) I just experienced this with the commtech project where I was listed as member but not as an admin. Loading the page listing instances gave a warning saying `Error: Unable to retriev... [15:50:19] 06Labs, 10Labs-Infrastructure: I/O on labmon1001 is very slow - https://phabricator.wikimedia.org/T127957#2199318 (10yuvipanda) :D Do either of you think you'll have the time to do the actually juggling around? [16:00:22] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#1766048 (10bd808) @demon and @mmodell: This is a project that I would very much like to make progress on as part of the #comm... [16:08:54] 06Labs, 10Tool-Labs: Make jsub / qsub default to trusty instances - https://phabricator.wikimedia.org/T94792#2199409 (10yuvipanda) Let's try this again, maybe. I'll send out a message saying this is going to happen in like a month. [16:09:07] 06Labs, 10Tool-Labs: Make jsub / qsub default to trusty instances - https://phabricator.wikimedia.org/T94792#2199412 (10yuvipanda) a:03yuvipanda [16:12:07] 06Labs, 10Tool-Labs, 13Patch-For-Review: Puppet fails on all Precise execution nodes - https://phabricator.wikimedia.org/T132282#2193357 (10yuvipanda) @Joe T94790 is the tracking ticket for getting rid of precise from tools. Unfortunately that's going to be a long hard slog, but I'm picking it up again to ma... [16:29:32] 06Labs, 10Tool-Labs, 13Patch-For-Review: Puppet fails on all Precise execution nodes - https://phabricator.wikimedia.org/T132282#2199495 (10valhallasw) Installed because of {T68354} [16:47:45] hey legoktm. around? [16:52:20] 06Labs, 10Tool-Labs, 10xTools-on-Labs: xtools-articleinfo spawning a large number of duplicate webservices - https://phabricator.wikimedia.org/T132471#2199614 (10yuvipanda) [16:58:54] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2199646 (10mmodell) bd808: I've added you to repository-admins group in case you need the permissions to test: https://phabr... [16:59:52] 06Labs, 10Tool-Labs, 06Zero: Tool labs tools should have a method of identifying Zero traffic - https://phabricator.wikimedia.org/T131934#2183516 (10yuvipanda) Awaiting more info from #zero folks. [17:00:31] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2199649 (10mmodell) One thing to consider is maybe running a separate phabricator instance just for this purpose. I can help... [17:01:26] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2199658 (10yuvipanda) I'd really want to not run another phab instance :D I don't think the labs team will have the bandwidth... [17:02:45] 06Labs, 10Tool-Labs, 10xTools-on-Labs: xtools-articleinfo spawning a large number of duplicate webservices - https://phabricator.wikimedia.org/T132471#2199661 (10Cyberpower678) The grid shouldn't be allowing duplicate tasks.~~~~ [17:14:06] 06Labs, 10Labs-Infrastructure: Disable creation of new Ubuntu Precise instances outside of whitelisted projects - https://phabricator.wikimedia.org/T102666#2199683 (10yuvipanda) [17:15:14] 06Labs, 10Labs-Infrastructure: Disable creation of new Ubuntu Precise instances outside of whitelisted projects - https://phabricator.wikimedia.org/T102666#2199687 (10Andrew) a:03Andrew [17:16:03] 06Labs, 10Tool-Labs, 10xTools-on-Labs: xtools-articleinfo spawning a large number of duplicate webservices - https://phabricator.wikimedia.org/T132471#2199690 (10yuvipanda) I've killed all jobs, and webservicemonitor has started one up. [17:22:29] 06Labs, 10Tool-Labs: Rewrite jsub in python - https://phabricator.wikimedia.org/T132475#2199735 (10yuvipanda) [17:22:38] 06Labs, 10Tool-Labs: Rewrite jsub in python - https://phabricator.wikimedia.org/T132475#2199748 (10yuvipanda) [17:25:48] (03PS83) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [17:26:04] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2199756 (10tom29739) If repositories get autocreated and tools do not use them then Diffusion will be filled with empty repos... [17:33:03] (03PS1) 10Yuvipanda: Add a warning to jsub if no release is specified [labs/toollabs] - 10https://gerrit.wikimedia.org/r/282975 [17:33:11] bd808: valhallasw`cloud ^ [17:33:48] is == even right? [17:34:17] I think it's == for numeric and eq for strings, but I don't know for sure >_< [17:34:34] yeah, that's what I thought as well [17:34:53] valhallasw`cloud: yup [17:35:09] (03CR) 10jenkins-bot: [V: 04-1] Add a warning to jsub if no release is specified [labs/toollabs] - 10https://gerrit.wikimedia.org/r/282975 (owner: 10Yuvipanda) [17:35:30] YuviPanda: + line 'Specify -l release=precise to suppress this warning' + a url on wikitech for more information? [17:35:35] 06Labs, 10Tool-Labs, 10xTools-on-Labs: xtools-articleinfo spawning a large number of duplicate webservices - https://phabricator.wikimedia.org/T132471#2199769 (10MusikAnimal) @yuvipanda Sorry! I don't know why it does that for -articleinfo. We use the exact same script on the base xtools project, and on -ec,... [17:36:31] valhallasw`cloud: sounds good. We can probably make a wiki page out of our etherpad... [17:36:36] and some more info [17:36:56] ^ YuviPanda [17:37:31] all the bash script does is `webservice restart`. Any idea why it's spawning new webservices rather than restarting the existing one? [17:37:59] MusikAnimal: thanks for your helpful response :) I am wondering - can you switch the webstart.sh to say 'try to access articleinfo, and if it times out, then restart'? [17:38:37] that's basically what the original webwatcher.sh did, and it had the same problem I believe [17:38:50] so I got rid of that and changed it a simple `webservice restart` [17:38:57] which seems harmless if done manually [17:39:11] (03CR) 10Ricordisamoa: "PS83 merges DragHelper.prototype.registerDraggables into DragHelper.prototype.getDraggable" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [17:39:29] MusikAnimal: I see. so I think the problem is that sometimes the qdel doesn't succeed, and restart somehow doesn't notice this and starts it again even though it shouldn't. [17:40:08] we'd be okay with a 5 second downtime, so many qdel any existing webservices then start one up? [17:41:14] MusikAnimal: try just using 'qstat | awk '{ print $1; }' | tail -1 | xargs qmod -rj' maybe? that just restarts the job in place with mechanisms that sge has. that should work most of the time. [17:41:23] YuviPanda: what's up? [17:41:46] legoktm: legobot is running on precise. Do you think you can move it to trusty? [17:42:22] MusikAnimal: we'll have a kubernetes backend to webservice soon, which will have better health monitoring. xtools-articleinfo migght be a good testcase whent hat becomes available [17:44:12] cool, I will change webstart.sh to use the script above and see if that does the trick [17:44:33] MusikAnimal: thanks! [17:44:44] note again that the issue of spawning new webservices doesn't seem to happen on the base xtools and -ec, for whatever reason, so I guess I'll leave those as-is [17:45:14] MusikAnimal: yeah. I think the way xtools-articleinfo seems to 'lock up' (where the process is running but not responding) might confuse SGE in some form... [17:45:18] YuviPanda: yeah, I'll just need to rebuild the venvs. Do you want me to do this ASAP or is end of the week okay? [17:45:24] legoktm: end of week is fine :) [17:46:11] YuviPanda: I see. Well I'll give this a go, and will monitor to see ensure new webservices aren't being created. Thank you!! [17:46:19] *to ensure [17:47:46] MusikAnimal: thanks :D [17:47:50] brb [17:50:01] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2199816 (10bd808) >>! In T117071#2199649, @mmodell wrote: > One thing to consider is maybe running a separate phabricator ins... [17:50:13] 06Labs, 10Tool-Labs, 10xTools-on-Labs: xtools-articleinfo spawning a large number of duplicate webservices - https://phabricator.wikimedia.org/T132471#2199817 (10MusikAnimal) Writing here to document what we're trying now: Per Yuvipanda's recommendation we're changing the restart script to use ``` qstat | aw... [17:52:26] MusikAnimal: or simpler: qmod -rj webservice-xtools-articleinfo [17:54:21] 06Labs, 07Documentation: Wikitech occasionally hides all your instances on Special:NovaInstance until you log out and log back in the wiki - https://phabricator.wikimedia.org/T132437#2199847 (10brion) [17:54:44] 06Labs, 07Documentation: Wikitech occasionally hides all your instances on Special:NovaInstance until you log out and log back in the wiki - https://phabricator.wikimedia.org/T132437#2198536 (10brion) @yuvipanda I owe you $20. This may be several beers or few, depending on what country we next meet in. ;) [17:56:44] i'll be happy when we ditch the remaining bits of the openstack control on wikitech.... was a clever system but the ui was always a bit funky :D [18:02:41] 06Labs, 07JavaScript: [jquery.chosen] Project filter widget missing the 'x' button on the tags in Special:NovaInstance etc - https://phabricator.wikimedia.org/T132480#2199882 (10brion) [18:08:10] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2199904 (10mmodell) > 8. Supports *blocking* post-receive hooks (eg. Make a HTTP call / run a command on post receive, and se... [18:11:49] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2199909 (10bd808) >>! In T117071#2199904, @mmodell wrote: > why post-receive? I don't think phabricator uses `post-receive` (... [18:15:47] hi all - anyone from wikimedia that has experience with large'ish scale git backend, we are looking into setups for osgeo.org; advice welcome [18:15:56] or others.. [18:16:45] darkblue_b: that's a... very generic question [18:16:55] darkblue_b: we use gerrit, and will be switching to diffusion at some point [18:18:01] ok valhallasw`cloud -- we have golang Gogs, GitLab and Kallithea setups now, evaluating ... [18:18:12] 06Labs, 10Labs-Infrastructure: Disable creation of new Ubuntu Precise instances outside of whitelisted projects - https://phabricator.wikimedia.org/T102666#2199930 (10Andrew) This is just $ glance image-update --is-public=false $ glance member-create tools $ glance member-create 06Labs, 10Labs-Infrastructure: Disable creation of new Ubuntu Precise instances outside of whitelisted projects - https://phabricator.wikimedia.org/T102666#2199935 (10Andrew) 05Open>03Resolved [18:29:23] 06Labs, 06Operations, 10procurement: Get an updated quote from HP for virt nodes - https://phabricator.wikimedia.org/T132363#2199968 (10Andrew) [18:30:12] 06Labs, 06Operations, 10hardware-requests, 10procurement: Get an updated quote from HP for virt nodes - https://phabricator.wikimedia.org/T132363#2195713 (10Andrew) [18:33:05] andrewbogott: You are aware that HP split into two companies last November, right? [18:33:34] 06Labs, 06Operations, 10hardware-requests: Get an updated quote from HP for virt nodes - https://phabricator.wikimedia.org/T132363#2199991 (10Krenair) [18:35:09] Matthew_: no, but mercifully I don't think I need to understand that in this case :) [18:35:31] andrewbogott: OK. Just understand they may send you to Hewlett Packard Enterprise for server stuff :) [18:35:59] It's all done via a reseller anyway [18:36:12] Okay. [18:37:04] Just making you aware :) [18:48:05] 06Labs, 06Operations, 10hardware-requests: Get an updated quote from HP for virt nodes - https://phabricator.wikimedia.org/T132363#2200095 (10Andrew) 05Open>03declined Declined in favor of proper procurement ticket T132485 [18:53:54] 06Labs, 10MediaWiki-General-or-Unknown, 07JavaScript: [jquery.chosen] Project filter widget missing the 'x' button on the tags in Special:NovaInstance etc - https://phabricator.wikimedia.org/T132480#2200137 (10matmarex) [18:53:56] darkblue_b: the folks who maintain our gerrit and diffusion infrastructure hang out in the #wikimedia-releng and #wikimedia-devtools channels [18:54:08] oh - ok thx [18:59:43] darkblue_b: any specific questions? I can answer questions about phabricator/diffusion/differential, and I've set up gitolite in the past [19:00:07] phabricator essentially does what gitolite does, which is itself a clone of what github does [19:00:50] ah thx twentyafterfour -- gitolite compare+contrast would be useful.. we wont be using Phab toolset, so differential probably wont help [19:00:56] 06Labs, 10Wikimedia-General-or-Unknown: [jquery.chosen] Project filter widget missing the 'x' button on the tags in Special:NovaInstance etc - https://phabricator.wikimedia.org/T132480#2200184 (10matmarex) Works for me. It loads the sprite image from https://wikitech.wikimedia.org/w/resources/lib/jquery.chosen... [19:04:10] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2200208 (10mmodell) >>! In T117071#2199816, @bd808 wrote: > The potential problem I see with running a separate instance is t... [19:06:25] darkblue_b: phabricator might still be a decent way of managing your repositories. It's essentially the same thing as gitolite but with a lot more automation around account management, key management and hooks... [19:06:58] so with gitolite you have to deal with account management yourself, phabricator integrates all of that into one package (which can still be somewhat automated / integrated with other tools) [19:07:10] gitolite is certainly a lot lighter weight [19:09:51] darkblue_b: so if you are comfortable with perl and don't mind doing a bit of integration work then gitolite is probably great. If you're more comfortable with PHP and want a more robust / complete solution, then phabricator might be for you. If you prefer ruby, then maybe gitlab... [19:16:52] 06Labs, 10Labs-Infrastructure: Disable creation of new Ubuntu Precise instances outside of whitelisted projects - https://phabricator.wikimedia.org/T102666#2200321 (10yuvipanda) Awesome, thanks! :) [19:19:49] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2200350 (10yuvipanda) >>! In T117071#2199909, @bd808 wrote: > I think the intent of a post-receive hook would be to build a n... [19:21:03] 06Labs, 07Documentation: Wikitech occasionally hides all your instances on Special:NovaInstance until you log out and log back in the wiki - https://phabricator.wikimedia.org/T132437#2200365 (10yuvipanda) [19:21:08] 06Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitech often loses track of internal openstack/nova session - https://phabricator.wikimedia.org/T101199#2200366 (10yuvipanda) [19:27:37] 06Labs, 10Tool-Labs, 07Tracking: Initial Deployment of Kubernetes to Tool Labs (Tracking) - https://phabricator.wikimedia.org/T111885#2200443 (10yuvipanda) [19:27:39] 06Labs, 10Labs-Sprint-115, 10Tool-Labs, 10labs-sprint-116, 10labs-sprint-118: Write admission controller disabling mounting of unauthorized volumes - https://phabricator.wikimedia.org/T112718#2200440 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Closed with https://phabricator.wikimedia.org/T116504... [19:28:06] 06Labs, 10Tool-Labs, 07Tracking: Initial Deployment of Kubernetes to Tool Labs (Tracking) - https://phabricator.wikimedia.org/T111885#1618964 (10yuvipanda) 05Open>03Resolved a:03yuvipanda I think this is done - despite DNS not working yet :) [19:30:17] ssh: Could not resolve hostname bastion1.wmflabs.org: No address associated with hostname [19:30:20] did I miss something? [19:30:48] YuviPanda: ^? [19:31:07] legoktm: bastion.wmflabs.org ? [19:31:17] legoktm: bastion1.wmflabs.org? I think that got deletd a long time ago... [19:31:40] Hmm [19:31:55] maybe it lingered on DNS until the recent switch? [19:32:04] either way, yeah, use bastion.wmflabs.org [19:32:06] heh, I still have *.pmtpa.wmflabs in my ssh config [19:32:07] ok [19:33:22] !fingerprints [19:33:22] ssh keys for bastion hosts: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [19:34:21] https://wikitech.wikimedia.org/w/index.php?title=Help%3ASSH_Fingerprints%2Fprimary.bastion.wmflabs.org&type=revision&diff=108036&oldid=105494 [19:34:53] heh [20:25:23] 06Labs: username case mismatch in keystone totp plugin - https://phabricator.wikimedia.org/T132455#2200778 (10Krenair) [20:29:32] did something happen to bastion-eqiad.wmflabs.org? [20:29:46] I'm having trouble ssh'ing to a labs instance [20:30:51] 06Labs: Figure out what to do about servicegrouphomedirpattern - https://phabricator.wikimedia.org/T125002#2200820 (10Andrew) 05Open>03Resolved I can't actually think of a better place for this setting -- it belongs in ldap along with other custom project settings. Would be nice if we abolished service grou... [20:30:53] 06Labs: Switch from ldap project assignment to keystone/mysql project assignment - https://phabricator.wikimedia.org/T124186#2200822 (10Andrew) [20:39:27] schana: no -- what's the issue? [20:40:22] I was getting a host name not resolved error, but upon changing my .ssh/config to be tools-login.wmflabs.org instead of bastion-eqiad.wmflabs.org, I was able to login [20:40:38] (after searching through email) [20:41:05] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2200881 (10mmodell) @yuvipanda: there is no reason we can't have a post-receive hook in phabricator, we would just need to in... [20:41:20] schana, that's a different host though [20:41:25] schana, try bastion.wmflabs.org [20:42:12] that works, Krenair [20:42:33] how long have you been using labs, schana? [20:42:52] if you started before the pmtpa migration that might explain why you have bastion-eqiad in there [20:42:59] not very, a couple months of accessing one host [20:44:29] Ah, yes, there is definitely a problem here though [20:44:44] dig bastion-eqiad.wmflabs.org @labs-ns0-former-placeholder.wikimedia.org returns bastion-eqiad.wmflabs.org. 3600 IN A 208.80.155.129 [20:44:58] dig bastion-eqiad.wmflabs.org @labs-ns0.wikimedia.org returns wmflabs.org. 3600 IN SOA labs-ns0.wikimedia.org. root.wmflabs.org. 1460486443 3600 600 86400 3600 [20:44:59] andrewbogott, ^ [20:47:44] !log tools.gerrit-reviewer-bot Moved GRB from Precise to Trusty (rebuilt venv in ~/venv, changed ~/gerrit_reviewer_bot.sh to match the new location) [20:47:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.gerrit-reviewer-bot/SAL, Master [20:47:53] Krenair: looking... [20:49:12] Krenair: is bastion-eqiad.mentioned in the docs someplace? [20:50:19] https://wikitech.wikimedia.org/wiki/Analytics/Onboarding#Sample_ssh_config is where I got my .ssh/config from initially [20:52:05] ugh [20:52:21] analytics should not have it's own replicas of docs from other places [20:58:08] YuviPanda, you made a bunch of C-records the other day, right? Can you see them in horizon? [20:58:11] (Can I?) [21:03:29] RECOVERY - Puppet run on tools-webgrid-generic-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [21:05:40] Krenair, schana, I created a cname for bastion-eqiad and also updated those docs to refer to 'primary.bastion.wmflabs.org' instead of bastion-eqiad [21:05:54] Krenair: let me fix that - I'll point it to this page? https://wikitech.wikimedia.org/wiki/SSH_access#SSH_configuration [21:06:59] madhuvishy: better yet, thanks [21:12:23] andrewbogott: np - done [21:35:45] !log rcm.cac updating all repos before testing a patch from gerrit [21:35:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm.cac/SAL, Master [21:53:06] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2201228 (10bd808) We obviously don't want to grant everyone with Tool Labs access direct membership in the repository-admins... [22:01:27] andrewbogott: I don't thikn we should do those. We should just have people refer to bastion.wmflabs.org [22:01:57] andrewbogott: I already changed https://wikitech.wikimedia.org/wiki/SSH_access [22:01:58] YuviPanda: that's fine, I was just trying to catch up with configs that already exist [22:02:14] YuviPanda: oh, you mean, don't use primary.bastion.wmflabs.org? [22:02:40] andrewbogott: no, I mean - it's ok to use primary.bastion.wmflabs.org. I just didn't want bastion-eqiad.wmflabs.org to exist :) [22:03:58] andrewbogott: I think we should just delete them again and tell people to use bastion.wmflabs.org. The documentation hasn't said bastion-eqiad.wmflabs.org for a long time now I think... [22:09:54] (03PS84) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [22:11:43] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2201369 (10yuvipanda) +1. I think this would also be a wonderful opportunity to allow people to select a LICENSE. [22:23:04] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:23:54] PROBLEM - Puppet run on tools-checker-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:23:56] PROBLEM - Puppet run on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:24:14] PROBLEM - Puppet run on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:24:45] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:25:23] PROBLEM - Puppet run on tools-redis-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:26:25] PROBLEM - Puppet run on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:26:49] PROBLEM - Puppet run on tools-exec-1404 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:27:41] PROBLEM - Puppet run on tools-exec-1408 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:27:55] PROBLEM - Puppet run on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:29:39] (03CR) 10Ricordisamoa: "PS84 fixes the enabling and disabling of the undo and redo buttons" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [22:30:30] PROBLEM - Puppet run on tools-exec-1405 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:59:10] RECOVERY - Puppet run on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:01:24] RECOVERY - Puppet run on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [23:02:52] RECOVERY - Puppet run on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [23:03:56] RECOVERY - Puppet run on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [23:05:29] RECOVERY - Puppet run on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [23:06:53] RECOVERY - Puppet run on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [23:24:11] 06Labs, 10Tool-Labs, 10Mail: correctly envelope forwarded email - https://phabricator.wikimedia.org/T120225#1848287 (10scfc) The name for this is "Sender Rewriting Scheme" (SRS); one solution is at https://github.com/Exim/exim/wiki/SRS. I think it should also be possible with pure `exim`, but I haven't look... [23:52:43] (03PS1) 10Jean-Frédéric: Add unit tests to test_update_database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/283108