[00:33:36] 06Labs, 10Continuous-Integration-Infrastructure, 07Wikimedia-Incident: Nodepool instance instance creation quota management - https://phabricator.wikimedia.org/T143016#2559620 (10thcipriani) Some random digging, there is some mention of "quota skew" in the #openstack-infra irc: http://eavesdrop.openstack.org... [05:44:25] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:54:56] PROBLEM - Puppet staleness on tools-exec-1211 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:59:00] PROBLEM - Puppet staleness on tools-exec-1213 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:59:10] PROBLEM - Puppet staleness on tools-exec-1204 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [06:26:09] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.76 ms [06:27:21] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 1.02 ms [06:28:09] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [06:34:36] PROBLEM - Host tools-secgroup-test-103 is DOWN: PING CRITICAL - Packet loss = 100% [06:52:26] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [06:55:54] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [07:57:11] (03CR) 10Jcrespo: [C: 04-1] "Do not mix encodings, assume you only have a limit of 3072 bytes for indexes and set the default database character set as utf8mb4; Leave " [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/305046 (https://phabricator.wikimedia.org/T142545) (owner: 10BryanDavis) [08:13:15] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [08:35:58] 06Labs, 10Tool-Labs: Puppet not running on tools-webgrid-lighttpd-1207 - https://phabricator.wikimedia.org/T143191#2560056 (10valhallasw) [08:42:28] 06Labs, 10Tool-Labs: Puppet not running on tools-webgrid-lighttpd-1207 - https://phabricator.wikimedia.org/T143191#2560078 (10valhallasw) Aha. Seems to be an ldap/networking issue, as around that time: ``` Aug 9 18:35:01 tools-webgrid-lighttpd-1207 CRON[17040]: Authentication failure Aug 9 18:39:01 tools-web... [08:50:00] 06Labs, 10Tool-Labs: Puppet not running on tools-webgrid-lighttpd-1207 - https://phabricator.wikimedia.org/T143191#2560085 (10valhallasw) But still: ``` Aug 17 08:45:33 tools-webgrid-lighttpd-1207 kernel: [ 0.000000] Initializing cgroup subsys cpuset ... Aug 17 08:45:33 tools-webgrid-lighttpd-1207 cron[930... [09:27:43] hello. is there a rate limit to API queries one can make from tool labs? [09:28:08] i'm getting "Could not resolve host: en.wikipedia.org" after some number of requests :( [09:31:51] samwilson: that's a dns issue, not a rate limit [09:31:57] on what host do you see that error? [09:39:04] 06Labs, 10Labs-Infrastructure, 10DBA: labsdb* has no automatic failover solution - https://phabricator.wikimedia.org/T141097#2560149 (10jcrespo) > can we cram them onto one new server I do not have right now exact numbers, and some may be excessively large, but yes, we can cram them in a single server, at l... [09:46:06] valhallasw`cloud: good point! this is on tools-bastion-03 [09:46:54] it does work most of the time though [09:51:48] samwilson: hrm. [09:53:49] samwilson: do you know whether this is a DNS server timeout or an error from the server? [09:54:01] (is there more information in an error log than just that could not resolve host?) [09:57:29] no, i'm looking to see what other verbose flags i can turn on to make it do it -- but then i'll have to run it again and hope it fails again [09:59:05] *nod* [09:59:09] dns issues are notoriously hard to debug [09:59:15] hmm [09:59:17] :( [10:00:18] so far, i've seen it with en.wikipedia.org and fr.wikisource.org [10:00:23] 06Labs, 10Tool-Labs: DNS resolution sometimes fails on tools-bastion-03 - https://phabricator.wikimedia.org/T143194#2560168 (10valhallasw) [10:05:06] 06Labs, 10Tool-Labs: DNS resolution sometimes fails on tools-bastion-03 - https://phabricator.wikimedia.org/T143194#2560198 (10Samwilson) Thanks @valhallasw. I raised T143193 earlier, because I thought this could be related to that, but it sounds like it's not likely to be. [10:17:04] It shouldn't really be doing a DNS request every time.... [10:17:13] Unless it's TTL related [10:18:58] User applications really shouldn't have to care about TTLs [10:20:42] doctaxon: please don't run heavy jobs on tools-bastion-03 [10:25:59] (03PS1) 10Volans: Add dummy Phabricator token for ops-monitoring-bot [labs/private] - 10https://gerrit.wikimedia.org/r/305214 (https://phabricator.wikimedia.org/T142085) [10:27:40] (03CR) 10Volans: [C: 032 V: 032] Add dummy Phabricator token for ops-monitoring-bot [labs/private] - 10https://gerrit.wikimedia.org/r/305214 (https://phabricator.wikimedia.org/T142085) (owner: 10Volans) [10:32:39] valhallasw`cloud: it never was a "heavy job", suddenly it is? [10:34:18] but okay, I took it down [10:35:00] doctaxon: it's a long running job that uses a full cpu [10:35:33] okay [10:35:38] no problem [10:35:43] thank you [10:35:48] yw [12:55:01] What's the difference between "debian-8.5-jessie (deprecated 2016-08-01)" and "debian-8.5-jessie" images on Labs? [13:00:51] tom29739: changes in labs infra, I think [13:01:13] It's likely that the deprecated version won't boot/puppet/etc correctly [13:01:35] While existing instances have been pushed changed config during a transition period [13:02:08] My instances are on the deprecated version. [13:05:28] As long as puppet runs regularly, that's not a problem [13:05:50] But if you create a new instance, use the non deprecated one [13:26:15] Hi, is somebody doing something with extdist.wmflabs.org? I'm trying to wget https://extdist.wmflabs.org/dist/extensions/VisualEditor-REL1_27-9da5996.tar.gz, I'm still receiving 502 Bad Gateway. [13:26:54] mail to @tools.wmflabs.org is not working right? [13:28:36] Nope. I'm trying to download https://extdist.wmflabs.org/dist/extensions/VisualEditor-REL1_27-9da5996.tar.gz and I'm receiving 502 Bad Gateway. [13:28:49] I didn't noted mailing to @tools.wmflabs.org. [13:32:16] mafk: Because of the domain I'm writing to labs channel. Should I ask somewhere else? [13:33:50] Urbanecm: sorry, I was asking a different question, not replying to yours; sorry for the confusion. [13:34:30] mafk: Oh. Thanks for explanation. [13:35:28] mafk: please file a bug about mail (including username/email address you're trying to use) [13:35:41] uitgevoerd [13:42:17] 06Labs, 10Labs-project-Extdist: http://extdist.wmflabs.org/ 502's (Bad Gateway) - https://phabricator.wikimedia.org/T143209#2560652 (10valhallasw) [13:47:28] valhallasw`cloud: Thanks for the report. Is there any other way to download packages as with ExtensionDistributor? [13:47:53] 06Labs, 10Labs-project-Extdist: http://extdist.wmflabs.org/ 502's (Bad Gateway) - https://phabricator.wikimedia.org/T143209#2560675 (10valhallasw) I'm having trouble reading the console log ("Your account is not in the project extdist.", even though it is), but I assume this is another instance of {T141673}. [13:48:13] Urbanecm: no idea, sorry [13:48:33] Okay. So I'll wait till this will be fixed. [15:02:47] 06Labs: cronspam from labscontrol1001, labstore1001, labnet1002.eqiad.wmnet, labsdb1003.eqiad.wmnet - https://phabricator.wikimedia.org/T132422#2560823 (10faidon) labtestservices2001 still warns daily about: `error: designate-common:1 duplicate log entry for /var/log/designate/designate-api.log` I think @Andrew... [15:12:43] 06Labs, 06Operations, 10Ops-Access-Requests: madhuvishy is moving to operations on 7/18/16 - https://phabricator.wikimedia.org/T140422#2560855 (10RobH) a:05madhuvishy>03MoritzMuehlenhoff Mortiz mentioned (in IRC a day or two ago) he would take care of the pwstore stuff, so I'm assigning this task to him. [16:18:02] (03PS10) 10BryanDavis: Add initial database schema [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/305046 (https://phabricator.wikimedia.org/T142545) [16:30:21] 10Tool-Labs-tools-Pageviews: Prioritize date labels where there are spikes in traffic - https://phabricator.wikimedia.org/T143227#2561254 (10MusikAnimal) [16:30:30] 10Tool-Labs-tools-Pageviews: Prioritize date labels where there are spikes in traffic - https://phabricator.wikimedia.org/T143227#2561268 (10MusikAnimal) p:05Triage>03Low [16:40:36] New instance creation fails in my project with "The requested instance cannot be launched as you only have 0 of your quota available." when the project is only using 5 of 8 instances. [16:41:14] RAM / CPU quotas? [16:43:49] tom29739: ^ [16:44:32] 06Labs, 10Phabricator, 07Puppet: Phabricator labs puppet role configures phabricator wrong - https://phabricator.wikimedia.org/T131899#2561286 (10demon) p:05Normal>03Low [16:45:04] yuvipanda, I'm in quota for those too. [16:45:49] in that case it's probably an openstack bug there's a ticket for somewhere, we're mostly just waiting on andrew to come back from vacation to look at it :( [16:47:28] can't seem to find it just now because grr [16:47:59] tom29739 https://phabricator.wikimedia.org/T142379 [16:49:33] Thanks. [16:50:15] I seem to be able to create instances on wikitech. [16:52:06] (03Draft2) 10Paladox: Replace git.wikimedia.org links with diffusion [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/305293 (https://phabricator.wikimedia.org/T139089) [16:52:10] (03Draft1) 10Paladox: Replace git.wikimedia.org links with diffusion [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/305293 (https://phabricator.wikimedia.org/T139089) [16:53:02] (03PS3) 10Paladox: Replace git.wikimedia.org links with diffusion [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/305293 (https://phabricator.wikimedia.org/T139089) [16:53:36] And security groups still don't work. [16:55:32] yeah, I think you should just use horizon for them [16:55:36] we'll probably remove them from the UI shortly [16:56:21] I'm using horizon. [16:56:35] I set them, but the ports don't open. [16:56:59] captcha-proxydb-01.privpol-captcha.eqiad.wmflabs [10.68.22.219] 5432 (postgresql) : Connection refused [16:57:16] But that port should be open. [16:57:23] aaah, ouch. I see. [16:57:42] wait you don't need to do security groups for things in the same project [16:57:53] assuming everything is part of the 'default' security group it should be able to see all the things in all the instances [16:58:22] also if the problem is security groups you usually get a timeout and it will seem to 'hang' - not get a connection refused [16:58:42] yep to all the above [17:01:53] (03PS4) 10Paladox: Replace git.wikimedia.org links with diffusion [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/305293 (https://phabricator.wikimedia.org/T139089) [17:11:39] yuvipanda, it doesn't seem to be doing that :/ [17:12:45] tom29739: what project, what instances, and what port from where to where [17:17:40] chasemp, project is privpol-captcha, instances are captcha-proxypostgres-01 and captcha-proxydb-01, trying to access port 5432 on both instances from captcha-apiproxy-02, bastion-01 and both instances [17:18:05] tom29739: ok I am tied up atm but I will take a look today [17:18:57] tom29739: likely postgres is just listening on localhost rather than on all ips? [17:19:48] It does that? [17:19:51] Oh. [17:20:04] tom29739: netstat shows 'tcp 0 0 localhost:postgresql *:* LISTEN' [17:20:28] yeah, that's standard for services that are typically only locally used [17:46:53] tom29739: solve your issue? [17:47:18] chasemp, yep [18:00:06] (03CR) 10Jcrespo: [C: 031] Add initial database schema [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/305046 (https://phabricator.wikimedia.org/T142545) (owner: 10BryanDavis) [18:01:46] (03CR) 10Jcrespo: "I will give you permissions, but you need to tell me which IP you will connect from; which in turn may (?) require iptables or vlan change" [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/305046 (https://phabricator.wikimedia.org/T142545) (owner: 10BryanDavis) [18:22:04] 06Labs: Investigate alternatives to NovaProxy - https://phabricator.wikimedia.org/T119936#1840467 (10AlexMonk-WMF) Ugh: https://github.com/containous/traefik/issues/362 I also wasn't able to deploy it without using their binary. They very much like Docker. [18:31:54] 06Labs: Investigate alternatives to NovaProxy - https://phabricator.wikimedia.org/T119936#2561629 (10AlexMonk-WMF) One feature I did like that Traefik advertised was built-in Let's Encrypt support... It can actually request a cert as the first TLS handshake for a given hostname comes in: https://docs.traefik.io/... [19:04:22] hi, I was wondering if it is possible to increase the amount of RAM assigned to a wmflabs instance? [19:06:51] Plato2000, which instance and why? [19:08:34] I am working on the drmf2016 instance, and we are trying to run a math search engine which uses BaseX at the same time as running the mediawiki service. It seems that if we have Apache running at the same time as the math search engine, the search engine process gets killed by the system at random points. [19:09:31] We suspected that this is because all of the available RAM is being used. [19:14:39] Plato2000: how big is that instance? [19:16:09] Luke081515: I'm not sure what you mean, but I know it has 4 GB of RAM [19:16:30] Plato2000: then you are using a medium instance [19:16:46] I think you need to create a new one, and then put the stuff on it [19:16:53] we have large (2x medium, so 8 gb ram) [19:17:02] and xlarge (4x medium, so 16 gb ram) [19:17:14] you can choose between these sizes, when you create the instance [19:17:36] Luke081515: Thanks for the information, I'll talk to the people in charge of this project [19:18:59] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Uzume was created, changed by Uzume link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Uzume edit summary: Created page with "{{Tools Access Request |Justification=I was considering a simple HTTP redirector service redirecting to WMF wikis based on a Wikidata entity and a sitelink key: e.g., Q515701..." [19:31:40] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Uzume was modified, changed by Uzume link https://wikitech.wikimedia.org/w/index.php?diff=817597 edit summary: [19:54:28] 06Labs, 10Labs-Infrastructure, 06Community-Tech-Tool-Labs, 10Striker, 03Scap3: Ensure that scap3 from tin can access californium - https://phabricator.wikimedia.org/T143253#2562015 (10bd808) [20:04:56] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Striker, and 2 others: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2562042 (10bd808) [20:05:00] 06Labs, 10Labs-Infrastructure, 06Community-Tech-Tool-Labs, 10Striker, and 2 others: Ensure that scap3 from tin can access californium - https://phabricator.wikimedia.org/T143253#2562039 (10bd808) 05Open>03Resolved a:03bd808 It looks like this should "just work" once all of the puppet rules are in pla... [20:06:33] (03CR) 10BryanDavis: "> I will give you permissions, but you need to tell me which IP you" [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/305046 (https://phabricator.wikimedia.org/T142545) (owner: 10BryanDavis) [20:08:40] (03CR) 10BryanDavis: "It would probably be good to allow the user with rights to create tables in the striker db to connect from terbium as well as californium." [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/305046 (https://phabricator.wikimedia.org/T142545) (owner: 10BryanDavis) [20:10:21] 06Labs: Get wmflabs.org on the Public Suffix List - https://phabricator.wikimedia.org/T142948#2562071 (10valhallasw) 05Open>03Resolved a:03yuvipanda One other relevant use case: letsencrypt rate limiting also uses the public suffix list, and (once the letsencrypt people deploy the latest version of their g... [20:23:35] 06Labs: Get wmflabs.org on the Public Suffix List - https://phabricator.wikimedia.org/T142948#2562124 (10AlexMonk-WMF) https://community.letsencrypt.org/t/how-often-is-the-internal-psl-updated/14240 [20:34:41] 06Labs: Request increased quota for Phlogiston labs project - https://phabricator.wikimedia.org/T143020#2554228 (10chasemp) There are currently no floating ip's assigned to this project. What is this going to be used for? We can do a temp bump for the shuffle for sure, but none of the metrics I can see make sen... [21:00:19] 06Labs, 06Operations, 13Patch-For-Review, 07Tracking: overhaul labstore setup [tracking] - https://phabricator.wikimedia.org/T126083#2562283 (10chasemp) [21:00:21] 06Labs, 06Operations: Create an NFS mount manager - https://phabricator.wikimedia.org/T140483#2562282 (10chasemp) 05Open>03Resolved [21:23:06] 06Labs, 10Tool-Labs: error starting webservice - https://phabricator.wikimedia.org/T142932#2562337 (10valhallasw) 05Resolved>03Open This also just happened for the `ptwikis` tool: ``` Traceback (most recent call last): File "/usr/bin/webservice-runner", line 27, in webservice.run(port) Fi... [21:26:25] 06Labs, 10Tool-Labs: error starting webservice - https://phabricator.wikimedia.org/T142932#2562344 (10valhallasw) ```` valhallasw@tools-webgrid-lighttpd-1207:/var/run$ ls -l lighttpd ls: cannot open directory lighttpd: Permission denied valhallasw@tools-webgrid-lighttpd-1207:/var/run$ ls -ld lighttpd drwxr-x--... [21:33:57] 06Labs: Request increased quota for Phlogiston labs project - https://phabricator.wikimedia.org/T143020#2562356 (10JAufrecht) There are two current instances: phab-01, which is phlogiston.wmflabs.org, and phab-02, which is phlogiston-dev.wmflabs.org. Phab-02 is fine, and could be scaled back from an xlarge to a... [21:35:48] 06Labs, 10Tool-Labs: error starting webservice - https://phabricator.wikimedia.org/T142932#2562357 (10valhallasw) Forcing a puppet run indeed displays: ``` Notice: /Stage[main]/Toollabs::Node::Web::Lighttpd/File[/var/run/lighttpd]/mode: mode changed '0750' to '1777 ``` maybe `/var/run/lighttpd` is not owned b... [21:37:26] 06Labs, 10Tool-Labs: Puppet not running on tools-webgrid-lighttpd-1207 - https://phabricator.wikimedia.org/T143191#2562373 (10valhallasw) I have disabled all queues on this host for now. [21:37:29] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 15User-bd808: Make jsub / qsub default to trusty instances - https://phabricator.wikimedia.org/T94792#2562374 (10bd808) Announced deprecation on labs-announce: https://lists.wikimedia.org/pipermail/labs-announce/2016-August/000144.html [21:43:15] RECOVERY - Puppet staleness on tools-webgrid-lighttpd-1207 is OK: OK: Less than 1.00% above the threshold [3600.0] [22:08:24] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Striker, and 2 others: Deploy "Striker" Tool Labs console to WMF production - https://phabricator.wikimedia.org/T136256#2562533 (10bd808) Scheduled production deployment for [[https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=817621&oldid=8... [22:11:59] 06Labs, 10Tool-Labs: error starting webservice - https://phabricator.wikimedia.org/T142932#2562563 (10valhallasw) I can't observe this on any other host: ``` valhallasw@tools-bastion-03:~/accountingtools$ for i in tools-webgrid-lighttpd-12{01..15}; do echo $i `ssh $i ls -ld /var/run/lighttpd`; done tools-webgr... [22:20:21] !log extdist rebooting extdist-01 T143209 [22:20:22] T143209: http://extdist.wmflabs.org/ 502's (Bad Gateway) - https://phabricator.wikimedia.org/T143209 [22:20:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Extdist/SAL, Master [22:22:18] legoktm: could you check the console log? [22:22:31] legoktm: to check whether it is https://phabricator.wikimedia.org/T141673 or not [22:22:33] uh, how? [22:23:03] (well, of -02, -01 is probably gone now, although maybe /var/log/syslog or dmesg has some info) [22:23:20] I don't think I can get into -02 [22:23:33] oh, console output works for me now [22:23:43] legoktm: you sshed into -01? ssh seemed dead to me [22:23:45] -01 is the main download server so I'd rather not have any downtime of it :S [22:23:51] no I rebooted from horizon [22:24:23] aaah [22:25:20] I've been leaving -02 in the broken state since it's the backup and so people can investigate, but -01 has to be working [22:25:20] legoktm: I'll try and see what I can find [22:25:24] thanks :) [22:25:49] legoktm: -02 can also be rebooted, I just had to check the console log [22:26:11] !log extdist rebooting extdist-02 T143209 [22:26:12] T143209: http://extdist.wmflabs.org/ 502's (Bad Gateway) - https://phabricator.wikimedia.org/T143209 [22:26:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Extdist/SAL, Master [22:26:22] valhallasw`cloud: I'm pulling up the message to verify but re: https://gerrit.wikimedia.org/r/#/c/305401/ it throws a "/opt/gridengine" deosn't exist warning [22:26:35] and our bind mount is /var/lib/gridengine [22:27:02] I'm not going to pretend to understand qconf internals but it's an environment thing for sure [22:27:46] 06Labs, 10Labs-project-Extdist: http://extdist.wmflabs.org/ 502's (Bad Gateway) - https://phabricator.wikimedia.org/T143209#2562677 (10valhallasw) -02 is definitely a hanging instance: ``` [321000.360669] INFO: task nscd:611 blocked for more than 120 seconds. [321000.361327] Not tainted 4.4.0-1-amd64 #1... [22:30:18] chasemp: added a comment to the python file and +1'ed [22:30:24] * valhallasw`cloud is off to bed [22:30:34] ah ok, later valhallasw`cloud! [22:34:42] ...and I broke it by doing that. Sorry :-( [22:35:38] heh [22:48:46] 06Labs, 10Tool-Labs: Make jsub warn when run without `-l release=...` - https://phabricator.wikimedia.org/T143282#2562755 (10bd808) [22:51:18] 06Labs, 10Tool-Labs: Make webservice warn when run with `-l release=precise` - https://phabricator.wikimedia.org/T143283#2562771 (10bd808) [22:51:41] 06Labs, 10Tool-Labs: Make jsub warn when run without `-l release=...` - https://phabricator.wikimedia.org/T143282#2562786 (10bd808) [22:59:35] 06Labs, 10Tool-Labs: Make trusty the default release target for jsub commands - https://phabricator.wikimedia.org/T143284#2562816 (10bd808) [23:00:33] 06Labs, 10Tool-Labs: Make jsub warn when run without `-l release=...` - https://phabricator.wikimedia.org/T143282#2562835 (10bd808) [23:00:36] 06Labs, 10Tool-Labs: Make trusty the default release target for jsub commands - https://phabricator.wikimedia.org/T143284#2562816 (10bd808) [23:04:11] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, 15User-bd808: Make jsub / qsub default to trusty instances - https://phabricator.wikimedia.org/T94792#2562868 (10bd808) [23:04:35] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from toollabs - https://phabricator.wikimedia.org/T94790#2562872 (10bd808) [23:08:01] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, 15User-bd808: Make jsub / qsub default to trusty instances - https://phabricator.wikimedia.org/T94792#2562885 (10bd808) [23:08:02] 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from toollabs - https://phabricator.wikimedia.org/T94790#2562886 (10bd808) [23:08:04] 06Labs, 10Tool-Labs: Make webservice warn when run with `-l release=precise` - https://phabricator.wikimedia.org/T143283#2562884 (10bd808)