[00:00:02] load-libraries should take care of that [00:00:36] I'm running arc liberate in phab/src [00:01:25] and that is taking forever [00:02:26] Negative24: fixed it - load-libraries just had the sprint extension listed, not the security extension [00:02:43] sudo bin/config set load-libraries '["/srv/phab/libext/Sprint/src","/srv/phab/libext/security/src"]' [00:02:53] what?! [00:03:20] I have set that a couple dozen times today [00:03:25] with puppet disabled [00:03:37] this is what I saw in https://phab-02.wmflabs.org/config/all/ load-libraries {"sprint":"/srv/phab/libext/Sprint/src"} [00:03:46] maybe puppet is overiding it [00:04:13] I ran puppet agent --disable before [00:05:23] Phab now reports the class in event listeners [00:06:38] I'm going to move the phlog to something other than the register function [00:09:08] Negative24: seems to be working, mostly anyway. I can't see the project because it's restricted but it did restrict the task that I just submitted [00:09:34] ok thats good. I guess I just need to refresh the existing tasks [00:10:33] twentyafterfour: thanks for your help. today wasn't a good day for me :) [00:10:56] Negative24: no problem, I'm sorry it's not been a good day. [00:11:26] I'm now going to write documentation like I always do when I figure stuff out [00:46:19] 10Wikimedia-Labs-wikitech-interface, 10Wikimedia-Site-requests, 5Patch-For-Review: Add "Nova Resource" to default search namespaces on wikitech.wikimedia.org - https://phabricator.wikimedia.org/T67132#1151870 (10Krenair) 5Open>3Resolved a:3Krenair [01:46:22] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Thibaut120094 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=150121 edit summary: [01:47:21] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/PhantomTech was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=150126 edit summary: [01:54:09] 10Tool-Labs: Tool Labs: Provide anonymized view of the user_properties table - https://phabricator.wikimedia.org/T60196#1152025 (10Ricordisamoa) [03:01:08] twentyafterfour: T493 is a bit confusing. Was this fixed with the phab update? [03:04:00] cause I can no longer reproduce [03:39:34] Krenair: When Help:Security_groups says "you can not remove or add security groups to an instance once it has been created." does it mean that groups and rules can't be changed or just groups? [03:39:52] no idea [03:39:54] sorry [03:40:31] np, chasemp ^ [03:46:18] 10Tool-Labs: Document how to turn shadow into master - https://phabricator.wikimedia.org/T91133#1152091 (10scfc) I don't want to appear dumber than I am :-), but I still only have a vague understanding from that page. Is the "shadow master watching the `heartbeat` file" service controlled by `service gridengine... [03:57:52] 10Tool-Labs: Harden mail server against incoming spam - https://phabricator.wikimedia.org/T67629#1152106 (10scfc) I think requiring valid DNS entries should be quite enough for a while. [04:01:05] where is everyone? [04:02:48] sleeping [04:03:44] but its only 9 PDT [04:04:45] and I'm on MDT [04:05:12] I'll ask tomorrow. For now I'm going to do the same thing. [04:21:15] 10Tool-Labs, 3ToolLabs-Goals-Q4: Provide a status page (list) of all active proxy definitions - https://phabricator.wikimedia.org/T88216#1152138 (10yuvipanda) There should be no stale proxy entries now tho - proxy listener is supposed to clean them up when the connection breaks... [04:36:37] 10Tool-Labs, 3ToolLabs-Goals-Q4: Provide a status page (list) of all active proxy definitions - https://phabricator.wikimedia.org/T88216#1152163 (10scfc) … except when the instance crashes due to an outage, for example :-) (currently "wikistream" has a proxy entry, but no web service running on `tools-webgrid-... [04:37:35] 10Tool-Labs, 3ToolLabs-Goals-Q4: Provide a status page (list) of all active proxy definitions - https://phabricator.wikimedia.org/T88216#1152165 (10yuvipanda) Ah of course :) Yeah if we can get epilogue scripts verifiably working I think we can switch to those :) [05:43:51] 10Tool-Labs, 3ToolLabs-Goals-Q4: The proxylistener service isn't puppetized - https://phabricator.wikimedia.org/T93121#1152274 (10yuvipanda) a:5scfc>3yuvipanda [05:47:51] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: The proxylistener service isn't puppetized - https://phabricator.wikimedia.org/T93121#1152280 (10yuvipanda) 5Open>3Resolved [05:49:01] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: The proxylistener service isn't puppetized - https://phabricator.wikimedia.org/T93121#1129895 (10yuvipanda) BTW, I've intentionally made it not be a subscribe or notify (I think / hope :P) - proxy listener should be restarted manually only, and with extreme... [05:49:33] YuviPanda: can wikishared DB accessiable via quarry? [05:49:53] kart_: quarry access everthing that’s accessible to labsdb [05:50:05] so you need to file a bug to make parts of it accessible to labsdb [05:50:26] YuviPanda: thanks! [06:01:34] 6Labs, 10Tool-Labs: Private SSL key got removed on tools-webproxy-01/tools-webproxy-02 and maybe other Labs instances as well - https://phabricator.wikimedia.org/T93212#1152297 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done for dynamicproxy-gateway. It shouldn't leave those machines, tho. [06:02:48] 10Tool-Labs-tools-Other: https://tools.wmflabs.org/merlbot-web/ 404s - https://phabricator.wikimedia.org/T85739#1152300 (10yuvipanda) [06:39:55] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [06:58:06] 10Tool-Labs, 5Patch-For-Review, 3ToolLabs-Goals-Q4: The portgranter service isn't puppetized - https://phabricator.wikimedia.org/T93120#1152393 (10yuvipanda) 5Open>3Resolved [07:04:54] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [07:57:00] 6Labs, 3ToolLabs-Goals-Q4: Allow labstores to hot or warm swap in case of failure - https://phabricator.wikimedia.org/T93589#1152408 (10yuvipanda) [08:00:01] hello [08:00:24] does anyone know if mail to toolserver.org is supposed to work? [08:03:05] nosy79: hi. it is, yeah - Coren put redirects in place, iirc [08:03:36] YuviPanda: ah ok thx then i will report a problem [08:04:06] nosy79: cool [08:05:12] 10Tool-Labs, 3ToolLabs-Goals-Q4: Put toolserver.org redirect configuration in git - https://phabricator.wikimedia.org/T85165#1152410 (10yuvipanda) [08:06:04] 6Labs, 6operations: Puppet failure on labstore1001 - https://phabricator.wikimedia.org/T92615#1152413 (10yuvipanda) @Coren this has been fixed, hasn't it? [08:06:53] 10Tool-Labs, 3ToolLabs-Goals-Q4: Monitor that the redundant webproxies have same state in terms of what they are proxying to whom - https://phabricator.wikimedia.org/T91484#1152414 (10yuvipanda) [08:12:04] 10Tool-Labs, 3ToolLabs-Goals-Q4: Have a 'undergoing scheduled maintenance' page for toollabs set up for scheduled maintenance - https://phabricator.wikimedia.org/T90595#1152418 (10yuvipanda) [08:35:55] good morning [08:36:26] 10Tool-Labs: Memory Exhausted Near / Tool labs error while querying with Python - https://phabricator.wikimedia.org/T93074#1152438 (10marcmiquel) Indeed, it broke: Traceback (most recent call last): File "cira_abroad.py", line 602, in langlist = [] File "/usr/lib/python2.7/dist-packages/MySQ... [09:25:06] 10Tool-Labs: E-Mail for toolserver.org - https://phabricator.wikimedia.org/T94011#1152463 (10Nosy) 3NEW a:3coren [09:38:07] am I the only one unable to log in to wikitech right now? [09:44:31] hey, i'm sending a query to the database which took almost 24h and it failed. [09:45:10] if anyone knows a better way to do it, i would sooo appreciate it [09:45:11] :) [09:45:14] SELECT rev_user_text, COUNT(*) FROM revision INNER JOIN page ON rev_page=page_id WHERE page_namespace=0 AND page_is_redirect=0 GROUP BY rev_user_text ORDER BY 2 DESC [09:45:46] it failed in the English Wikipedia, obviously [09:50:08] YuviPanda: any idea? [10:07:30] 10Tool-Labs: Cannot start java processes using the grid engine - https://phabricator.wikimedia.org/T69588#1152534 (10Nosy) The script exits via jsub now with: tools.toolserverdb@tools-bastion-01:~$ tail cron-tools.toolserverdb-2.err Exception occurred while VM initialising. java/lang/NoClassDefFoundError: java/... [11:01:22] (03CR) 10Revi: [C: 031] Adding new tags for #wikimedia-commons-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/199242 (owner: 10Steinsplitter) [11:54:50] Is there anyone available that can look at and close [[meta:Requests for comment/Abandoned Labs tools]] today, please? [11:54:58] @link [11:54:58] https://meta.wikimedia.org/wiki/Requests_for_comment/Abandoned_Labs_tools [11:55:41] Thanks [12:33:48] 6Labs, 6operations: Puppet failure on labstore1001 - https://phabricator.wikimedia.org/T92615#1152771 (10coren) 5Open>3Resolved a:3coren Puppet applies cleanly, and that error is a DNS fail resolving 'puppet' so I'm a little mystified on how that could even have happened in prod. [12:47:43] 10Tool-Labs: E-Mail for toolserver.org - https://phabricator.wikimedia.org/T94011#1152800 (10Nosy) Now I get a connection but I may not send mail. : host toolserver.org[208.80.155.197] said: 550 Administrative prohibition (in reply to RCPT TO command) Reporting-MTA: dns; monoro.de X-Po... [12:50:15] 10Tool-Labs, 3ToolLabs-Goals-Q4: Add monitoring to toolserver.org redirects - https://phabricator.wikimedia.org/T94022#1152804 (10coren) 3NEW [13:08:17] 10Tool-Labs: E-Mail for toolserver.org - https://phabricator.wikimedia.org/T94011#1152842 (10coren) Logs show the last email to be forwarded dates from 2015-03-23 06:26:09 at which point the filesystem filled because the apache logs (for the web redirects) grew by nearly 21G in less that 7 hours, faster than log... [13:09:19] 10Tool-Labs: E-Mail for toolserver.org - https://phabricator.wikimedia.org/T94011#1152845 (10coren) >>! In T94011#1152800, @Nosy wrote: > Now I get a connection but I may not send mail. You caught me while I was in the middle of restarting the service. :-) [13:09:42] 10Tool-Labs: E-Mail for toolserver.org - https://phabricator.wikimedia.org/T94011#1152846 (10Nosy) I have mail now :) [13:10:02] 10Tool-Labs: E-Mail for toolserver.org - https://phabricator.wikimedia.org/T94011#1152853 (10Nosy) 5Open>3Resolved [13:12:26] 10Tool-Labs: E-Mail for toolserver.org - https://phabricator.wikimedia.org/T94011#1152858 (10yuvipanda) All of this should be puppetized as well :) we can keep the email data private but let's puppetized everything else. [13:20:22] 10Tool-Labs-tools-Other: Migrate http://toolserver.org/~dispenser/* to Tool Labs - https://phabricator.wikimedia.org/T68868#1152876 (10coren) [13:20:24] 10Wikimedia-Labs-Other, 6operations, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1152877 (10coren) [13:20:24] 10Wikimedia-Labs-General: Make user_email_authenticated status visible on labs - https://phabricator.wikimedia.org/T70876#1152873 (10coren) 5Open>3declined a:3coren Our requirement is "visible to users without special rights", which seems to apply here - except that this is only exposed for editors that ha... [13:27:19] 10Wikimedia-Labs-General: Make user_email_authenticated status visible on labs - https://phabricator.wikimedia.org/T70876#1152889 (10Krenair) Isn't this something the API exposes as the emailable property in ApiQueryUsers? [13:28:06] 10Wikimedia-Labs-General: Make user_email_authenticated status visible on labs - https://phabricator.wikimedia.org/T70876#1152890 (10Krenair) Either way, CCing @csteipp since there's questions over whether this should really be exposed or not. [13:29:24] 10Wikimedia-Labs-General: Make user_email_authenticated status visible on labs - https://phabricator.wikimedia.org/T70876#1152892 (10coren) Not quite. Emailable is a composite property: email is confirmed **and** not disabled by the user. Exposing user_email_authenticated shows that the editor has supposed and... [13:36:50] 6Labs: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1152900 (10coren) I'm going to place this on our "sprint" for next week (Mar 30) [13:38:25] 6Labs: Set storage service up in codfw - https://phabricator.wikimedia.org/T85605#1152904 (10coren) 5Open>3Resolved This is now complete, although there is no Labs to serve //to// in codfw. Labstore2001 will nevertheless be the destination of the backup of eqiad's labstores. [13:38:26] 6Labs, 5Patch-For-Review: Replicate data between codfw and eqiad - https://phabricator.wikimedia.org/T85606#1152906 (10coren) [13:38:27] 6Labs: Labs available in the new data centre (with Neutron/IPv6) - https://phabricator.wikimedia.org/T85609#1152908 (10coren) [13:38:28] 6Labs: Storage capacity & redundancy expansion (tracking) - https://phabricator.wikimedia.org/T85604#1152907 (10coren) [13:41:57] 10Wikimedia-Labs-Infrastructure, 10Beta-Cluster, 7Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#1152917 (10coren) This is "fixed" only insofar as there is now more /var to fill before things break; but that provides no guard against... [13:43:54] 6Labs, 10Beta-Cluster, 6operations: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#1152920 (10coren) a:5coren>3None The new partitioning scheme has more room in /var for stray core dumps; though this does not address the necessity of cleaning/collecting them as apropr... [13:45:30] 6Labs, 10Wikimedia-Labs-Infrastructure, 5Patch-For-Review: Internal DNS look-ups fail every once in a while - https://phabricator.wikimedia.org/T72076#1152924 (10coren) a:5coren>3None >>! In T72076#1115332, @Dzahn wrote: > could this have caused T92351 ? No, that's dnsmasq being a bad DNS server in gen... [13:47:44] 6Labs, 10Wikimedia-Labs-Infrastructure: Set up second DNS server for Labs instances - https://phabricator.wikimedia.org/T90234#1152927 (10coren) 5Open>3Resolved a:3coren This is made moot by the impending arrival of a true pdns for Labs managed via Designate (see T87280) [13:48:47] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1152934 (10coren) 5Open>3Resolved This has been worked around in beta, and the new DNS server (see T... [14:17:46] 6Labs: Ganglia broken for labstore1001 (again) - https://phabricator.wikimedia.org/T92618#1152998 (10coren) 5Open>3Resolved This is working now, and has been for a while; I don't know whether it's possible to alert when ganglia doesn't receive data because, like Yuvi, I monitor labs through graphite because... [14:48:55] Can I set the "default wiki” to something else on my labs-vagrant host? [14:49:48] I looked at MWMultiVersion and it seems to want the wiki’s DB name to appear in the URL before the @base_domain [14:50:18] werdna: You can change that in hiera. Not well documented [14:50:36] I can’t figure out what the hiera variable is called [14:50:42] * bd808 looks for a host where he has done this [14:51:13] I think I had it set up but then I ran ‘git clean’ and deleted the local.yaml [14:51:18] so I’m trying to rebuild this list :( [14:52:07] My SUL testing host has this local.yaml -- https://dpaste.de/Ujep [14:52:50] I don’t really see how that would help me [14:52:56] I want to use the ‘livingstyleguide’ wiki as the default [14:53:00] https://sulfinalization.wmflabs.org hits the "main" wiki and then https://login-sulfinalization.wmflabs.org hits the login wiki [14:53:37] ah. gotcha [14:53:38] hmm, but what if I want to ignore the ‘main’ wiki and just send everybody to the ‘livingstyleguide’ wiki? [14:53:51] reviews on https://gerrit.wikimedia.org/r/#/c/184891/ asked me to put the livingstyleguide role on a different wiki [14:54:01] I could always not use the role [14:54:09] or use a special Hiera variable to put it somewhere else [14:54:16] or install a real wiki instead of using labs-vagrant [14:55:27] If you set mediawiki::multiwiki::base_domain to ".wmflabs.org" then "livingstyleguide.wmflabs.org" would work [14:56:09] I guess that makes sense [14:58:18] Alternately some heira magic could be added to the role to let it be attached to the default wiki with local config [14:58:48] I've been meaning to review that mw-v patch and some others but haven't found time yet [14:59:35] yeah, as always :) [15:00:33] hrr [15:00:33] Notice: /Stage[main]/Apache/File[/etc/apache2/sites-enabled/00-lsg-redirect.conf]/ensure: removed [15:01:56] bd808: actually, out of interest, is there a way to get all the MW puppet classes without using labs-vagrant? [15:02:00] s/classes/modules [15:02:15] I was thinking about your comments about vagrant being mainly a dev environment [15:02:34] was wondering how easy it would be to set up an alternative system for deploying MW to a server [15:04:11] The puppet modules are just in the mediawiki-vagrant git repo. They aren't split out into individual repos per module or anything [15:04:21] but where do they come from? [15:04:30] were they written specifically for vagrant? copied from operations/puppet.git? [15:04:38] https://github.com/wikimedia/mediawiki-vagrant [15:04:46] written specifically for mw-v [15:05:03] often with bits and pieces copied from operations/puppet [15:05:14] there are a few submodules that are shared with both [15:05:24] hmm [15:05:57] The mediawiki module is all custom pretty much [15:06:19] and others are usually greatly simplified [15:06:27] nod [15:06:40] "reuseable" puppet modules are not something we typically do here [15:06:45] :D [15:06:47] They are fragile and ugly [15:07:39] I ask partly because I think there’s a general use case for “Give me a box with MediaWiki and all the trimmings, but extensible in a way other than submitting patchsets to MediaWiki-Vagrant”, and also because I’m working on puppetising a client site [15:07:55] *nod* [15:08:24] I think this is basically the "third-party vagrant" use case [15:08:52] How can we make a better-than-a-tarball distribution basically [15:09:53] not only third party, but also for random dev wikis like the living style guide and other stuff on labs [15:10:37] There's an idea from Ori in the backlog on phabricator for a web gui to build a tailored distribution [15:10:52] * werdna looks [15:11:01] backlog for which team? [15:11:09] mw-vagrant [15:11:36] Something like https://puphpet.com/ [15:11:40] got it https://phabricator.wikimedia.org/T53782 [15:23:09] bd808: one more thing… can I “include this puppet code” into vagrant somehow… should I just create a new role and not submit it? [15:35:38] werdna: Yeah, there's a place built for that actually. Make your modules in puppet/modules/local and then apply them by adding the module name to a "classes" list in the local.yaml hiera file [15:35:45] I have slides somewhere .... [15:35:56] bd808: \o/ [15:35:57] thanks! [15:36:30] werdna: That use case is covered in this presentation -- https://www.mediawiki.org/wiki/File:MediaWiki-Vagrant_Tech_Talk_2014Q4.pdf [15:36:48] There's even a video you can watch if you're bored :) [15:38:14] Slides starting at https://www.mediawiki.org/w/index.php?title=File:MediaWiki-Vagrant_Tech_Talk_2014Q4.pdf&page=20 talk about a local puppet class [15:51:31] hi all, if I need to get a gerrit repository for developing and deploying an extension, is this the right place to request for one - https://www.mediawiki.org/wiki/Gerrit/New_repositories [15:53:03] codezee: yes [15:53:30] codezee: although i've found that it's usually easier to start the project on github, and then use that page to request the github repo be migrated to gerrit at a later point [15:53:50] codezee: that way you don't have to wait around for the repository to be created before you can start hacking [15:54:54] codezee: and i like to think that it helps the maintainers because it's clear that this repo will have actual code in it, and isn't just a random idea that the requester will never get around to writing code for. [15:55:53] cscott: could you tell what is the average time after the request that we get access, in case of migration from github to gerrit? [15:56:52] codezee: well, you get access right away on github, and you can just 'git push' the latest code from github to gerrit whenever the repo is created, so there's no delay at any point. [15:58:34] codezee: but you can look at the history for https://www.mediawiki.org/wiki/Git/New_repositories/Requests to get an idea for typical turnaround time. [15:59:51] sorry, I mean https://www.mediawiki.org/w/index.php?title=Git/New_repositories/Requests/Entries&action=history (there is some tricky transclusion going on on the /Requests page) [16:00:52] QChris seems to handle new repo requests within a day or so. [16:04:28] cscott: does starting a project on github mean pushing basic code on a repo created on our own github-user? [16:04:42] codezee: yes. [16:06:00] codezee: then when you fill out the request, specify the github url as the "existing repository elsewhere you want imported" [16:08:01] cscott: since I need to start work on an extension, I guess it makes sense to first setup extension:BoilerPlate on my repo and then request access for gerrit [16:08:12] cscott: thanks! for the help :) [16:08:19] codezee: no problem! [16:21:26] cscott: just a last question, is the request for migration of a git-repo to gerrit made from the same page - https://www.mediawiki.org/wiki/Gerrit/New_repositories [16:22:00] codezee: yes, when you click the 'make request' button you'll see the text about "existing repository elsewhere you want imported" [16:36:02] So Help:Access says that security groups can't be changed. Does that apply to groups and rules or just groups? [16:36:22] would I be able to change rules in a group still? [16:36:37] *Help:Security_groups [16:55:57] Negative24: Rules in groups can be changed; you just can't change the groups that applies to an instance [16:56:28] ok. so I can add a rule to a group that's already applied to an instance and it will apply that rule? [16:57:03] Coren: so its really only a problem if you have to change which instances get different policies after they have been created [16:57:19] Right. [16:57:28] good [17:27:39] bd808: I didn't know about vagrant share. Neat! [17:27:57] it's pretty cool [17:27:59] I've been using ngrok (and set up my own wildcard subdomain that points to it) [17:28:40] E.g. Livingstyleguide.test.andrewjgarrett.com -- dnsmasq'd to 127.0.0.1 locally, but in my DNS it points to ngrok. [17:28:49] I think it's a better solution, but it took me a bit to set up :p [17:33:22] twentyafterfour: would a diffusion phabricator for T560 be worthy of a new phab instance? it would need port 222 open. [17:33:38] or just change phab-01 or phab-02? [17:43:28] (03PS1) 10Southparkfan: Use always metawiki.labsdb server [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/199947 [17:45:58] (03CR) 10John F. Lewis: [C: 032 V: 032] Use always metawiki.labsdb server [labs/tools/WMT] - 10https://gerrit.wikimedia.org/r/199947 (owner: 10Southparkfan) [18:00:06] (03CR) 10Legoktm: [C: 04-1] "See inline comments. At what point is this channel going to be a duplicate of -multimedia? :/" (032 comments) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/199242 (owner: 10Steinsplitter) [18:00:30] seems https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database is out of date - how do i connect to the mysql db - i'm on deployment-db2 (or should i switch hosts?) [18:05:04] `sql foowiki` [18:06:51] legoktm, see -dev [18:07:10] ah [18:32:41] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Luke081515 was created, changed by Luke081515 link https://wikitech.wikimedia.org/wiki/Nova+Resource%3aTools%2fAccess+Request%2fLuke081515 edit summary: Created page with "{{Tools Access Request |Justification=I want to create a bot on the german wikipedia (maybe later on other projects), and I need tools to make the bot run, when i'm not at my..." [18:38:31] on deployment-bastion (and, I suppose, tin) should /srv/mediawiki-staging be owned by mwdeploy—is there any harm in that? Currently, jenkins can't pull in changes if those changes create a file, dunno when that started. [18:38:48] see: https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/2184/console [18:57:58] 6Labs: Sync up the new labs NFS project filesystem with the live one - https://phabricator.wikimedia.org/T93792#1154025 (10coren) This is taking slightly longer than expected, with the maps project being the stalling point. The copy should be over in a few hours, I'll advise at that time whether it is still a r... [19:18:01] (03Abandoned) 10Steinsplitter: Adding new tags for #wikimedia-commons-tech [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/199242 (owner: 10Steinsplitter) [19:41:09] chasemp: https://gerrit.wikimedia.org/r/#/c/199966 [20:27:37] 6Labs, 5Patch-For-Review: Write a custom designate-sink handler - https://phabricator.wikimedia.org/T93928#1154429 (10Andrew) Attached patch does the first two, and reverse dns. It does not do the ec2-id entry, and most likely never will -- we should rip out use of that as much as possible. [20:33:52] 10Wikimedia-Labs-General: Make user_email_authenticated status visible on labs - https://phabricator.wikimedia.org/T70876#1154490 (10csteipp) I don't think users expect for the date to be visible, but neither is there an attack (that I'm aware of) that exposing the date opens up. So it's never been worth fixing... [20:40:20] 10Tool-Labs, 6Security: Toollabs project database accounts are not password protected - https://phabricator.wikimedia.org/T59067#1154608 (10csteipp) [20:40:21] 10Wikimedia-Labs-General, 6Security: Data leakage user table "new" databases like wikidatawiki_p and the wikivoyage databases - https://phabricator.wikimedia.org/T56847#1154609 (10csteipp) [20:40:33] 10MediaWiki-extensions-OpenStackManager, 6Security: OpenStackManager API modules not requiring tokens - https://phabricator.wikimedia.org/T51785#1154618 (10csteipp) [20:40:43] 10Wikimedia-Labs-Other, 6Security: http://creatorlinks.wmflabs.org cross site scripting - https://phabricator.wikimedia.org/T46620#1154628 (10csteipp) [20:40:44] 10Wikimedia-Labs-bots, 6Security: http://bots.wmflabs.org cross site scripting - https://phabricator.wikimedia.org/T46618#1154627 (10csteipp) [21:14:02] 10Tool-Labs: Unattended upgrades are failing from time to time - https://phabricator.wikimedia.org/T92491#1155076 (10scfc) ``` From: root@tools.wmflabs.org (Cron Daemon) Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) To: root@tools.wmflabs.org Da... [22:03:11] 6Labs, 5Patch-For-Review: Write a custom designate-sink handler - https://phabricator.wikimedia.org/T93928#1155338 (10Andrew) 5Open>3Resolved [22:03:11] 6Labs, 5Patch-For-Review: Investigate replacing our custom DNS code with Designate - https://phabricator.wikimedia.org/T87280#1155340 (10Andrew) [23:26:08] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1155635 (10scfc) 5Resolved>3declined That may be, but this is certainly not //resolved//, and whethe... [23:28:31] 6Labs, 10Wikimedia-Labs-Infrastructure: Set up second DNS server for Labs instances - https://phabricator.wikimedia.org/T90234#1155646 (10scfc) 5Resolved>3declined As with T92351, for this task to be //resolved//, there would need to exist two DNS servers for Labs instances. [23:30:38] 6Labs, 10Wikimedia-Labs-Infrastructure, 10Continuous-Integration, 6operations: dnsmasq returns SERVFAIL for (some?) names that do not exist instead of NXDOMAIN - https://phabricator.wikimedia.org/T92351#1155653 (10coren) That may still be an option, once he have at least //one// that actually works right....