[01:46:49] 06Labs, 06Operations, 13Patch-For-Review: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm - https://phabricator.wikimedia.org/T132216#2274128 (10bd808) [02:23:15] quiddity: hmm I need to take better care of it... [02:23:18] (Quarry) [02:23:23] tyvm for the restart, Krenair [02:23:30] Krenair: usually just restarting celery makes it work... [02:23:37] labs-morebots: do you still need a grrit-wm restart? [02:23:37] I am a logbot running on tools-exec-1214. [02:23:37] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [02:23:38] To log a message, type !log . [02:27:16] Luke|away: ^ [04:26:59] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [04:30:03] looking [04:33:30] !log tools reboot tools-worker-1004, lots of ksoftirqd stuckness despite no actual containers running [04:33:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [04:38:15] PROBLEM - SSH on tools-worker-1004 is CRITICAL: Connection refused [04:40:44] RECOVERY - Free space - all mounts on tools-worker-1004 is OK: OK: tools.tools-worker-1004.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) tools.tools-worker-1004.diskspace._var_lib_docker.byte_percentfree (More than half of the datapoints are undefined) [04:48:17] 06Labs: tools-worker-1004 is dead - https://phabricator.wikimedia.org/T134717#2274210 (10yuvipanda) [06:55:18] PROBLEM - Puppet run on tools-worker-1008 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:30:21] RECOVERY - Puppet run on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [07:48:47] 10Wikibugs, 10grrrit-wm, 07TestMe: fix grrrit-wm bug channels - https://phabricator.wikimedia.org/T95578#2274552 (10Legoktm) @Danny_B: why did you tag this as #testme? [07:49:36] 10Wikibugs, 10grrrit-wm, 07TestMe: fix grrrit-wm bug channels - https://phabricator.wikimedia.org/T95578#1194962 (10yuvipanda) What does this task mean? [07:50:59] 10Wikibugs, 10grrrit-wm, 07TestMe: fix grrrit-wm bug channels - https://phabricator.wikimedia.org/T95578#2274556 (10valhallasw) 05Open>03Invalid Not something I can remember. I think this was a reminder 'we should make sure all channels match up', but the 'send a patch and we'll merge it' approach has ta... [07:52:21] !log ores manually restarted nginx load balancer (ores-lb-02) It wasn't needed since puppet did it already. [07:52:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [08:12:16] (03PS1) 10Lokal Profil: Limit language codes to lower case or hyphen [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287573 (https://phabricator.wikimedia.org/T134728) [08:16:39] YuviPanda, yeah, I tried that first :/ [08:18:11] Krenair: hmm I need to do a massive upgrade of all the things there as well [08:18:14] Quarry needs more love :| [08:18:17] no py3 [08:18:19] old celery versions... [08:31:25] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2274655 (10jcrespo) Today at 8:30: ``` MariaDB m5-master (none) > select * FROM information_schema.table_statistics WHERE table_schema='pdns'; +--------------+-----------------+-----------... [08:32:42] (03CR) 10Lokal Profil: "I would say that for our target audience/likely source of contributions MediaWiki standard is probably the way to go =/" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287290 (owner: 10Lokal Profil) [08:35:55] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2274668 (10jcrespo) labtestservices2001 is still replicating from m5-master. Next steps? [08:39:58] 10Wikibugs, 10grrrit-wm, 07TestMe: fix grrrit-wm bug channels - https://phabricator.wikimedia.org/T95578#2274672 (10Legoktm) It was that the channels bugs for #grrrit-wm are sent to should match the channels bugs in #wikibugs are sent to :) [09:04:16] (03CR) 10Jean-Frédéric: [C: 032] Limit language codes to lower case or hyphen [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287573 (https://phabricator.wikimedia.org/T134728) (owner: 10Lokal Profil) [09:05:14] (03CR) 10jenkins-bot: [V: 04-1] Limit language codes to lower case or hyphen [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287573 (https://phabricator.wikimedia.org/T134728) (owner: 10Lokal Profil) [09:08:46] (03CR) 10Jean-Frédéric: "Recheck" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287573 (https://phabricator.wikimedia.org/T134728) (owner: 10Lokal Profil) [09:14:53] 10Wikibugs: wikibugs is reporting column changes again - https://phabricator.wikimedia.org/T134733#2275224 (10Legoktm) [09:17:31] (03PS1) 10Legoktm: Column changes are now "core:columns" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/287577 (https://phabricator.wikimedia.org/T134733) [09:18:12] (03CR) 10Legoktm: [C: 032] Column changes are now "core:columns" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/287577 (https://phabricator.wikimedia.org/T134733) (owner: 10Legoktm) [09:18:44] (03Merged) 10jenkins-bot: Column changes are now "core:columns" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/287577 (https://phabricator.wikimedia.org/T134733) (owner: 10Legoktm) [09:19:45] !log tools.wikibugs legoktm: Deployed 1715d33c22249a3f1dde01d6eec929b3ad6abb8c Column changes are now "core:columns" wb2-phab [09:19:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [09:21:28] 10Wikibugs, 13Patch-For-Review: wikibugs is reporting column changes again - https://phabricator.wikimedia.org/T134733#2275224 (10Legoktm) 05Open>03Resolved a:03Legoktm ^ caused no notifications in IRC. [09:32:14] 10Labs-Other-Projects: Update http://orgcharts.wmflabs.org - https://phabricator.wikimedia.org/T660#2275298 (10Qgil) 05stalled>03declined Thank you @Krenair for the link / update. I'm declining this task after interpreting the reality of this project. Feel free to reopen if anyone disagrees. [10:07:29] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Decide on upgrade policy for Kubernetes - https://phabricator.wikimedia.org/T133598#2275340 (10yuvipanda) Monday / Tuesday? [10:23:14] (03CR) 10Jean-Frédéric: [C: 031] Limit language codes to lower case or hyphen [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287573 (https://phabricator.wikimedia.org/T134728) (owner: 10Lokal Profil) [10:23:22] (03CR) 10Jean-Frédéric: [C: 032] Limit language codes to lower case or hyphen [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287573 (https://phabricator.wikimedia.org/T134728) (owner: 10Lokal Profil) [10:24:13] (03Merged) 10jenkins-bot: Limit language codes to lower case or hyphen [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287573 (https://phabricator.wikimedia.org/T134728) (owner: 10Lokal Profil) [10:57:03] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Write a k8s admission controller to enforce that all containers running come from our private repository - https://phabricator.wikimedia.org/T133515#2275459 (10yuvipanda) Nagf has been moved as well. [10:58:19] !ping [10:58:19] !pong [11:15:41] !log tools.heritage Deployed latest from Git: 4f82b32 (T134728) [11:15:43] T134728: matchWikiprojectLink assumes language codes contain word characters when they may also contain dash - https://phabricator.wikimedia.org/T134728 [11:15:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL, Master [11:28:16] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Write a k8s admission controller to enforce that all containers running come from our private repository - https://phabricator.wikimedia.org/T133515#2275537 (10yuvipanda) [11:28:18] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Use a 'pause' container from our private repo, not gcr.io - https://phabricator.wikimedia.org/T133873#2275535 (10yuvipanda) 05Open>03Resolved All containers in our cluster now are from docker-registry.tools.wmflabs.org \o/ [11:44:47] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Provision a .kube/config file for all tools - https://phabricator.wikimedia.org/T133999#2275644 (10yuvipanda) Small python script that does the following: 1. Look at all tool users in LDAP 2. Check if there's an entry in token auth for all of them 3. For the users tha... [11:54:05] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Setup PAM inside containers used in Tool Labs - https://phabricator.wikimedia.org/T134748#2275666 (10yuvipanda) [11:55:04] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Setup PAM inside containers used in Tool Labs - https://phabricator.wikimedia.org/T134748#2275682 (10yuvipanda) [11:55:06] 06Labs, 10Tool-Labs: Build containers for use with Tool Labs - https://phabricator.wikimedia.org/T130668#2275681 (10yuvipanda) [12:23:41] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Setup PAM inside containers used in Tool Labs - https://phabricator.wikimedia.org/T134748#2275755 (10yuvipanda) This should also run without nscd / nslcd caching daemons. [12:26:38] 10Wikibugs, 10grrrit-wm, 07TestMe: fix grrrit-wm bug channels - https://phabricator.wikimedia.org/T95578#2275765 (10Danny_B) >>! In T95578#2274552, @Legoktm wrote: > @Danny_B: why did you tag this as #testme? To find out if the problem still persist or the task can be closed... >>! In T95578#2274556, @valha... [12:49:51] General question: If I'm using the same labs instance every time, is the internal IP then stable? (the 10.68.... IP which is visible if I'm using a bot, which is not logged in) [13:14:00] Luke081515: yes essentially it is assigned for the life of the VM I believe "static dhcp" [13:39:09] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 13Patch-For-Review: Use a 'pause' container from our private repo, not gcr.io - https://phabricator.wikimedia.org/T133873#2247942 (10chasemp) https://groups.google.com/forum/#!topic/google-containers/jVjv0QK4b_o [13:44:47] chasemp: thx [14:29:15] YuviPanda: Can you sync me at some point wrt https://phabricator.wikimedia.org/T133515 - e.g. where is stuff now and how do I update it (or a link to docs :) ) [14:41:00] 06Labs: new bootstrap-vz jessie images don't log (and maybe don't start at all) - https://phabricator.wikimedia.org/T133551#2276243 (10Andrew) p:05Triage>03High [14:41:20] 06Labs: Update custom sink handlers for Designate Mitaka - https://phabricator.wikimedia.org/T134280#2276244 (10Andrew) p:05Triage>03Normal [15:09:21] 06Labs: image rsync from labcontrol1001 to labcontrol1002 never cleans up - https://phabricator.wikimedia.org/T134769#2276398 (10Andrew) [15:09:51] 06Labs: image rsync from labcontrol1001 to labcontrol1002 never cleans up - https://phabricator.wikimedia.org/T134769#2276413 (10Andrew) https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/openstack/manifests/glance/service.pp;7865bec468b6e42fadcc5a6e409e4294b3ffac0e$109 [15:21:32] 10Labs-project-extdist, 10Continuous-Integration-Config, 13Patch-For-Review, 07WorkType-Maintenance: Move labs/tools/extdist tox tests to jessie - https://phabricator.wikimedia.org/T102894#2276480 (10hashar) [15:39:48] !log deployment-prep migrating deployment-flourine to labvirt1009 [15:39:49] Please !log in #wikimedia-releng for beta cluster SAL [15:39:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, dummy [16:34:14] that bot is supper smart [17:00:06] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs: bibleversefinder tool broken - https://phabricator.wikimedia.org/T134786#2277029 (10Dzahn) [17:00:31] that bibleversefinder tool looks broken, doesnt it [17:00:39] fe. http://tools.wmflabs.org/bibleversefinder/?book=Zechariah&verse=12:11&src=! [17:00:48] those links appear in WP articles all over the place [17:01:01] so end users notice [17:01:15] got a mail to dns-admin@ from one of them [17:01:41] need to contact the tool developer, not much we can do (since the webservice is up and running) [17:02:40] 10Tool-Labs-tools-Other: bibleversefinder tool broken - https://phabricator.wikimedia.org/T134786#2277048 (10yuvipanda) Someone should contact the tool maintainer (has only one maintainer :() at https://wikitech.wikimedia.org/wiki/User:Atethnekos [17:03:57] YuviPanda: cool, getting the right contact is a major step [17:04:05] replying to the reporter [17:04:12] thanks mutante [17:04:30] mutante: I just went to tools.wmflabs.org and ctrl-f'd [17:05:11] (03PS14) 10Yuvipanda: Rewrite jsub in python [labs/toollabs] - 10https://gerrit.wikimedia.org/r/285435 (https://phabricator.wikimedia.org/T132475) (owner: 10BryanDavis) [17:05:28] (03CR) 10Yuvipanda: [C: 032] "Rejoice!" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/285435 (https://phabricator.wikimedia.org/T132475) (owner: 10BryanDavis) [17:09:56] 06Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2277092 (10Andrew) labtestservices uses a totally different openstack install (with different domains and hosts and so on) so it never really made sense to replicate from m5-. So, I would... [17:20:55] PROBLEM - Host tools-worker-1011 is DOWN: PING CRITICAL - Packet loss = 100% [17:37:47] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Setup PAM inside containers used in Tool Labs - https://phabricator.wikimedia.org/T134748#2277229 (10yuvipanda) I realized after reading through more documentaiton that I don't actually want PAM but I want NSS. [17:51:59] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 13Patch-For-Review, 15User-bd808: Rewrite jsub in python - https://phabricator.wikimedia.org/T132475#2277246 (10bd808) As noted in the commit message of the patch the port makes a few small changes to the output that is sent to qsub: * `-j yes|no` flag alw... [18:22:11] 06Labs, 10MediaWiki-extensions-OATHAuth, 06Security-Team, 10wikitech.wikimedia.org, 13Patch-For-Review: wikitech 2fa provisioning form does so without confirmation - https://phabricator.wikimedia.org/T130892#2277362 (10dpatrick) a:05dpatrick>03csteipp [18:24:38] 06Labs, 10Tool-Labs, 06Operations: toolserver.org certificate to expire 2016-06-30 - https://phabricator.wikimedia.org/T134798#2277377 (10chasemp) [18:25:35] 06Labs, 10Tool-Labs, 06Operations: toolserver.org certificate to expire 2016-06-30 - https://phabricator.wikimedia.org/T134798#2277394 (10valhallasw) We still provide redirects, and I think those are still used. Can we use the new letsencrypt-for-simple-frontends manifests to keep this online? [18:36:53] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 13Patch-For-Review, 15User-bd808: Rewrite jsub in python - https://phabricator.wikimedia.org/T132475#2199735 (10scfc) >>! In T132475#2277246, @bd808 wrote: > As noted in the commit message of the patch the port makes a few small changes to the output that... [19:20:09] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 13Patch-For-Review, 15User-bd808: Rewrite jsub in python - https://phabricator.wikimedia.org/T132475#2277598 (10bd808) The omission of `-l h_vmem=X` is only for user supplied `h_vmem` settings. A better way to state the change would probably be that explic... [19:57:11] (03PS1) 10BryanDavis: www: Guard against undefined index errors [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287690 [19:57:13] (03PS1) 10BryanDavis: www: make some performance optimizations [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287691 [19:57:35] chasemp: ^ a couple of patches that might make tools.admin a bit better [19:58:16] sweet, let me dig myself out of a hole here and I'll take a peek [19:59:06] (03CR) 10Merlijn van Deen: [C: 032] "lgtm" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287690 (owner: 10BryanDavis) [19:59:39] chasemp: shall I also do the performance one or are you doing that one already? [20:00:03] please do :) [20:00:11] I'll just poke about and see if I understand then [20:04:22] (03CR) 10Merlijn van Deen: [C: 04-1] "Looks OK, but I'm not sure about the 301 vs 302. Are we sure we want to redirect all ?[A-Z]* to wikitech forever?" (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/287691 (owner: 10BryanDavis) [20:04:42] * valhallasw`cloud goes back into hiding [20:04:48] valhallasw`cloud: good question I guess [20:05:01] I'm not necessarily against a 301 [20:05:46] but given that the entire URL structure is a bit odd, we might decide to change it at some point [20:06:00] all of the actions in tools.admin are lower case as are the static assets. The ?[A-Z] is for ?Privacy, ?Help, ?Rules today [20:06:15] *nod* [20:06:44] except https://tools.wmflabs.org/Tool_Labs_logo_thumb.png [20:07:05] ah, but that doesn't sue the query string so that's OK [20:07:27] and then some others actually use the /admin prefix, such as https://tools.wmflabs.org/admin/libs/jquery.js [20:08:05] the /admin/ gets normalized out [20:08:16] $uri = preg_replace( '@^/admin@', '', $uri ); [20:10:02] * valhallasw`cloud nods [20:10:17] A bigger perf fix would be to change the default behavior from ?list to something lighter like a new static text welcome screen [20:10:32] that sounds totally reasonable to me [20:11:18] multi-pane front panel: link to wikitech, link to hay's list thingie, link to {all tools | grid status | etc} [20:11:38] yeah that would be pretty easy to do [20:11:39] and maybe links to the N most popular tools [20:11:55] heh. if we only knew what those would be [20:12:58] For N=1: geohack ;-) [20:13:33] (but that's mostly by pageviews, and that's because it's linked/semi-embedded on many wiki pages, I think) [20:14:47] anyhow, coming back to the 301. I'm fine with either; if you feel the 301 is the cleanest option, let's go for that. [20:15:19] the 301 on the /tool bounces is more important [20:15:38] small input: ?list as the homepage probably does cost us a lot [20:15:43] Is loading JQuery & Google stuff allowed on Labs? If not what should I do if I find any? And citations please ;-) [20:15:58] chasemp: yeah. I'm going to follow up with some change to handle that [20:16:30] Dispenser: best doc right now is https://phabricator.wikimedia.org/T129936 [20:16:40] Dispenser: no in general not good practice and check out http://tools.wmflabs.org/static-browser/ [20:17:05] as for fixing, a phab ticket is a good start as well as adding as a blocker at https://phabricator.wikimedia.org/T133919 [20:18:29] valhallasw`cloud: Is ReleaseTaggerBot unwell? https://phabricator.wikimedia.org/p/ReleaseTaggerBot/ … [20:18:36] The thing is I always see "Privacy Violation", but can't find it in the privacy policy [20:18:47] valhallasw`cloud: Possibly the rollback to wmf.23 broke it? [20:18:57] Dispenser: *nod* we are working with WMF legal to clarify that. [20:19:57] James_F: sorry, I just turned off my computer [20:20:20] Can you send me an email so I don't forget tomorrow? [20:20:23] T129936 is about fixing the ambiguity in the Labs TOU [20:20:23] T129936: Ensure that Terms of Use document restrictions on third-party web interactions - https://phabricator.wikimedia.org/T129936 [20:20:24] valhallasw`cloud: No worries. Will do. [20:20:56] e.g. the way its written Google Analytics is completely allowed (privacy policy forbids third-party cookies only, not first party) [20:22:25] yep that's why we are working on clarifying the language, but the idea is to stem the case of 3rd party resources called from tool labs that make assurances of privacy moot [20:22:29] afaiu [20:22:52] The argument that I have been making is that any non-informed interaction with a non-WMF server should be forbidden [20:23:41] So links are fine obviously, but loading images, js, css, etc from an external server should not be allowed [20:46:31] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 13Patch-For-Review, 15User-bd808: Rewrite jsub in python - https://phabricator.wikimedia.org/T132475#2277960 (10scfc) Then I misunderstood your earlier comment: I thought that you meant that if a user does not pass `-mem X` to `jsub`, `jsub` will not call... [21:19:55] trying to get a self hosted puppet instance running on labs - but the puppet run keeps failing at the install puppetmaster stage 'Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install puppetmaster' [21:25:42] madhuvishy, try installing puppetmaster manually? [21:25:45] just the package [21:26:44] yea, apt-get install puppetmaster , see what happens [21:26:56] then run puppet again [21:34:47] that seems to be working [21:35:25] :) [21:35:25] thanks Krenair mutante! [23:01:59] 06Labs: new bootstrap-vz jessie images don't log (and maybe don't start at all) - https://phabricator.wikimedia.org/T133551#2278827 (10Andrew) ok, it was a long time getting there but I now have a pull-request ready, here: https://github.com/andsens/bootstrap-vz/pull/320 [23:10:02] (03PS1) 10Jean-Frédéric: Strip wikitext comments out of parsed values in templates [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/287792 (https://phabricator.wikimedia.org/T134727)