[00:06:24] !log deployment-prep Synchronized portals: https://gerrit.wikimedia.org/r/#/c/250851/ [00:06:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [00:24:45] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Antoine2711 was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=199442 edit summary: [00:45:51] andrewbogott: want me to kill the jobs on 1203 now? [01:32:58] 10Quarry: Add 'download in HTML format' option (Quarry) - https://phabricator.wikimedia.org/T117644#1780562 (10XXN) 3NEW [01:35:18] 10Quarry: Add 'download as wikitable' option (Quarry) - https://phabricator.wikimedia.org/T117645#1780573 (10XXN) 3NEW [01:39:11] 10Quarry: Add chat or forum (Quarry) - https://phabricator.wikimedia.org/T117647#1780591 (10XXN) 3NEW [01:42:20] 10Quarry: Add chat or forum (Quarry) - https://phabricator.wikimedia.org/T117647#1780601 (10Legoktm) You can use IRC. [01:43:19] 10Quarry: Add chat or forum (Quarry) - https://phabricator.wikimedia.org/T117647#1780604 (10yuvipanda) There's no Quarry specific IRC channel, and I think there should be a link to whatever it is from Quarry itself. I'll also try to get a Flow board created for this. That seems useful. [01:43:49] lolol [01:44:11] legoktm: ? [01:45:31] I suggested people use IRC :P [01:46:06] IRC is way too ephemeral to be useful [01:46:10] for something as low volume as quarry [01:46:46] re: ephemeral: http://bots.wmflabs.org/~wm-bot/logs/ [01:47:14] sigh [01:47:23] very user friendly >_> [01:47:35] ok, I'm not going to have this discussion [01:48:27] legoktm: hmm, so https://www.mediawiki.org/wiki/Talk:MediaWiki is flow [01:48:32] but most talk pages on mw.org aren't? [01:48:34] (03PS1) 10Legoktm: Send all User-* projects to /dev/null [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/250893 [01:49:00] YuviPanda: if they used to be LQT, they're now Flow. You can ask Roan or James to convert random pages [01:49:07] legoktm: ah ok! [01:49:11] but only on mw.org, I guess? [01:49:20] for meta I need to provide human sacrifices, I suppose [01:49:27] Yeah, only mw.o [01:49:30] ok [01:51:56] legoktm: done, thanks [02:05:46] 6Labs, 6operations: Investigate whether to use Debian's jessie-backports - https://phabricator.wikimedia.org/T107507#1780643 (10ori) >>! In T107507#1534816, @coren wrote: > Opsen consensus on IRC is that jessie-backports should be disabled fleet-wide and any needed package brought into jessie-wikimedia. Why?... [03:49:08] 6Labs, 10Tool-Labs, 10Diffusion, 3labs-sprint-119: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#1780732 (10yuvipanda) Ok, summarizing discussions between me, @mmodell and @demon: # We can use diffusion for this! # There's an unstalbe API that @demon is... [04:30:20] (03CR) 10Legoktm: [C: 032] Send all User-* projects to /dev/null [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/250893 (owner: 10Legoktm) [04:30:51] (03Merged) 10jenkins-bot: Send all User-* projects to /dev/null [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/250893 (owner: 10Legoktm) [04:31:29] !log tools.wikibugs Updated channels.yaml to: 775987cc6b7998d7495fcae652546ab2df0d1d6a Send all User-* projects to /dev/null [04:31:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [05:12:00] (03PS1) 10KartikMistry: labs: Set JWT token for Beta [labs/private] - 10https://gerrit.wikimedia.org/r/250906 (https://phabricator.wikimedia.org/T116134) [06:04:54] kart_: that isn't actually a valuble secret, right? [06:08:02] YuviPanda: no. it is different than production. [06:08:11] not valuble. [06:09:01] ok [06:09:14] (03CR) 10Yuvipanda: [C: 032 V: 032] labs: Set JWT token for Beta [labs/private] - 10https://gerrit.wikimedia.org/r/250906 (https://phabricator.wikimedia.org/T116134) (owner: 10KartikMistry) [07:41:42] (03Abandoned) 10Legoktm: Use RotatingFileHandler for logging, set to ~100MB [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/181433 (owner: 10Legoktm) [08:41:36] 6Labs, 10Wikimedia-Labs-General, 10Datasets-Archiving, 10Datasets-General-or-Unknown, 5Patch-For-Review: Make pagecounts-all-sites available in Wikimedia Labs - https://phabricator.wikimedia.org/T93317#1780996 (10ArielGlenn) 5Open>3Resolved a:3ArielGlenn done. [09:04:28] 10Quarry: Add 'download in HTML format' option (Quarry) - https://phabricator.wikimedia.org/T117644#1781007 (10Aklapper) Please provide a specific usecase example describing what you would like to accomplish. [09:04:32] 10Quarry: Add 'download as wikitable' option (Quarry) - https://phabricator.wikimedia.org/T117645#1781008 (10Aklapper) Please provide a specific usecase example describing what you would like to accomplish. [09:47:58] 6Labs, 6Phabricator: [Phabricator Labs] Ops Access Request don't work - https://phabricator.wikimedia.org/T117664#1781091 (10Luke081515) 3NEW [09:48:08] 6Labs, 6Phabricator: Bugs at security options at phabricator labs - https://phabricator.wikimedia.org/T117665#1781100 (10Luke081515) 3NEW [09:48:17] 6Labs, 6Phabricator: [Phabricator Labs] Ops Access Request don't work - https://phabricator.wikimedia.org/T117664#1781108 (10Luke081515) [09:48:21] 6Labs, 6Phabricator: Bugs at security options at phabricator labs - https://phabricator.wikimedia.org/T117665#1781107 (10Luke081515) [09:54:02] 6Labs, 10Beta-Cluster-Infrastructure, 10Labs-Infrastructure, 7Graphite, 7Shinken: Delete more specific deployment-prep graphite datapoints - https://phabricator.wikimedia.org/T111540#1781112 (10fgiunchedi) diamond discovers the mount point locally and starts pushing metrics for those [10:01:41] 6Labs, 6Phabricator: [Phabricator Labs] Ops Access Request don't work - https://phabricator.wikimedia.org/T117664#1781142 (10Aklapper) That's because Herald rules are not synced from the Phabricator production instance to random Labs instances. So I don't consider this a bug per se. [10:09:01] 6Labs, 6Phabricator: [Phabricator Labs] Ops Access Request don't work - https://phabricator.wikimedia.org/T117664#1781150 (10Luke081515) https://phab-01.wmflabs.org/herald/rule/1/ exists. And I got an own instance with that too, and imported the actual security code from the wikimedia git, so the code is uniqu... [11:11:14] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1781285 (10fgiunchedi) 3NEW [11:43:45] 10Tool-Labs-tools-Other, 6Community-Tech: Edit Count tool on labs shows text in wrong language - https://phabricator.wikimedia.org/T111512#1781388 (10Qgil) Just for context: why is this task in your radar? @aklapper, any ideas of how to help fixing it? I was thinking that perhaps it could help to send an emai... [12:03:14] 10Tool-Labs-tools-Other, 6Community-Tech: Edit Count tool on labs shows text in wrong language - https://phabricator.wikimedia.org/T111512#1781470 (10Aklapper) This task was initially filed under #Community-Tech so if #Community-Tech is not going to take it I propose declining it here in Phabricator. https://... [12:14:39] 10Tool-Labs-tools-Other, 6Community-Tech: Edit Count tool on labs shows text in wrong language - https://phabricator.wikimedia.org/T111512#1781485 (10Qgil) OK, thank you for the extra background. CCing some contributors to that GitHub repo, ping that at least one of them is interested in volunteering as Google... [14:27:28] 6Labs, 6Phabricator, 7Tracking: Bugs at security options at phabricator labs - https://phabricator.wikimedia.org/T117665#1781746 (10Negative24) [14:29:13] 6Labs, 6Phabricator: [Phabricator Labs] Ops Access Request don't work - https://phabricator.wikimedia.org/T117664#1781753 (10Negative24) Well the security extension isn't even present nor configured on Phab-01 (which creates the Herald rules). Known issue. [15:35:55] 10Tool-Labs-tools-Other, 6Community-Tech: Edit Count tool on labs shows text in wrong language - https://phabricator.wikimedia.org/T111512#1781905 (10Krenair) >>! In T111512#1781470, @Aklapper wrote: > https://github.com/x-Tools/xtools/issues/60 is the canonical bug report. Krenair commented how to probably fi... [15:40:34] 6Labs, 10Labs-Infrastructure, 6operations: deployment tracking of codfw labs test cluster - https://phabricator.wikimedia.org/T117097#1781916 (10Papaul) [15:40:36] 6Labs, 10Labs-Infrastructure, 6operations, 10ops-codfw, 5Patch-For-Review: on-site tasks for labs deployment cluster - https://phabricator.wikimedia.org/T117107#1781914 (10Papaul) 5Open>3Resolved This task is complete. closing it [15:42:06] 10Tool-Labs-tools-Other, 6Community-Tech: Edit Count tool on labs shows text in wrong language - https://phabricator.wikimedia.org/T111512#1781918 (10Aklapper) Ahem. Yeah, sorry. I'll stick to writing Kr*n*** from now on. :) [15:53:58] (03PS1) 10Greg Grossmeier: Add #Testing-Initiative-2015 to -releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/250978 [16:00:41] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1781998 (10scfc) Is the instance still inaccessible after the first Puppet run (30 minutes)? Otherwise this looks like a duplicate of T110304. [16:43:31] (03CR) 10Dduvall: [C: 031] Add #Testing-Initiative-2015 to -releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/250978 (owner: 10Greg Grossmeier) [16:43:49] (03CR) 10Legoktm: [C: 032] Add #Testing-Initiative-2015 to -releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/250978 (owner: 10Greg Grossmeier) [16:44:21] (03Merged) 10jenkins-bot: Add #Testing-Initiative-2015 to -releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/250978 (owner: 10Greg Grossmeier) [16:44:57] !log tools.wikibugs Updated channels.yaml to: 9a7f239ec5c34604d9e48901fc28e997ea53a5e4 Add #Testing-Initiative-2015 to -releng [16:44:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [17:03:15] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1782334 (10fgiunchedi) yep still inaccessible, console log keeps repeating like below. also T110304 seems a duplicate of T103808 ? ```lines=5 [1;35merr: Could not request certificate: Connec... [17:47:15] 6Labs, 6operations: Investigate whether to use Debian's jessie-backports - https://phabricator.wikimedia.org/T107507#1782752 (10coren) >>! In T107507#1780643, @ori wrote: > Why? Is the chat log public, by any chance? Sorry, if I recall correctly most of it was held on an unlogged channel. The discussion at h... [18:19:42] Would anyone like to provide an explanation? [18:19:44] https://en.wikipedia.org/w/index.php?title=Airport_Express&action=info <= No disambig page_props [18:19:45] select * from page_props WHERE pp_page=268058 <= No disambiguation page prop [18:19:47] https://en.wikipedia.org/w/api.php?action=parse&page=Airport_Express <= Has the property [18:55:49] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1783095 (10yuvipanda) @fgiunchedi was this created with the same name of an instance that was deleted a little while ago? [18:56:35] on an older instance i created there was a /dev/mapper/vd-second--local--disk, i don't remember explicitly creating this but perhaps i did. On a new instance this isnt there, is the correct way to then use lvcreate to make one? [18:57:44] ebernhardson: there's a 'lvm::srv' role or something you can turn on [18:58:42] YuviPanda: thanks trying now [18:58:44] Dispenser: I suggest filing a bug about it with the disambiguation extension? [18:59:46] I might be related to other oddities I've seen https://phabricator.wikimedia.org/T117696 [19:02:53] possibly [19:03:04] I think it was an 'expensive' query page and only ran once a month or somesuch [19:05:24] Dispenser, that might be the 5th time I've seen this bug created :) [19:05:26] There was a mass update via Template:Disambiguation last month [19:08:09] The "fun" part is I got the before and after api parse query from last month [19:15:41] Is anyone here familiar with WMF-wiki/Tool Labs CORS requirements? [19:21:16] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1783323 (10fgiunchedi) >>! In T117673#1783095, @yuvipanda wrote: > @fgiunchedi was this created with the same name of an instance that was deleted a little while ago? the name was used befor... [19:23:03] fhocutt: hm. I thought requests to api.php were OK as long as they were only GET, but my test seems to suggest otherwise [19:23:13] var invocation = new XMLHttpRequest(); invocation.open('GET', 'https://nl.wikipedia.org/w/api.php?action=query', true); invocation.onreadystatechange=alert; invocation.send(); [19:23:17] ^ in the javascript console [19:23:52] that onreadystatechange doesn't work, I think, but it gives me 'XMLHttpRequest cannot load https://nl.wikipedia.org/w/api.php?action=query. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://developer.mozilla.org' is therefore not allowed access.' [19:23:56] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1783347 (10fgiunchedi) p:5High>3Normal \o/ using another name works, I guess this is known? `filippo@filippo-test-precise2:~$ ` [19:24:04] I'm modifying the Citation Bot gadget to have an API, so I want to load a response from it [19:24:41] http://tools.wmflabs.org/citations-dev/fakeresult.php?title=query is what I have so far, and I put Access-Control-Allow-Origin on the server side [19:24:44] as for using data /from/ tool labs, the tool should set the relevant CORS headers [19:24:56] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1783355 (10yuvipanda) Yeah that error message is consistent with some component somewhere not caring about DNS TTL and just caching the old name forever, preventing puppet from running. @andr... [19:25:36] fhocutt: yes, that works. [19:25:39] so the really weird thing is that this works when the call happens automatically, but when we put it into onclick for a button it gets CORS errors again. [19:25:44] var invocation = new XMLHttpRequest(); invocation.open('GET', 'https://tools.wmflabs.org/citations-dev/fakeresult.php?title=query', true); invocation.send(); [19:25:49] on a non-toollabs page [19:25:52] https://test.wikipedia.org/w/index.php?title=User:Fhocutt_%28WMF%29/common.js&action=edit [19:25:54] line 33 [19:27:29] mmmm, I don't see the button [19:27:31] when we forgot the quotation marks and it ran all the time, it worked [19:27:43] oh, ugh, that's because it's on my common.js [19:27:47] valhallasw`cloud: You'll have to copy over the script. [19:27:48] oh, there, next to the save page [19:27:50] yeah [19:27:57] yeah, I copied it over, but just didn't see the button :) [19:28:14] oh, yeah [19:28:16] ...the button submits for me? [19:28:27] valhallasw`cloud: Check logs? [19:28:46] It does a console log, which gets blocked due to CORS. [19:28:55] fhocutt: ok, if I change it to 'button' instead of 'submit', it works for me [19:29:00] This is fake article text This is a fake article summary [19:29:03] ...weird. [19:29:30] ...so those are different I guess? [19:30:10] submit also works, if I check preserve log [19:31:06] what's preserve log? [19:31:28] a checkbox to prevent the console log to be cleared when navigating to a different page [19:31:30] might be chrome-specific [19:31:40] oh, could be, I'm on firefox [19:32:52] well, I am super confused, but that works now! Thanks so much. [19:36:15] (03PS9) 10ArthurPSmith: Added a Wikidata-based "chart of the nuclides" under /nuclides [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 [19:36:41] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1783481 (10scfc) Does it become accessible after a reboot? (I //think// I successfully launched a Precise instance in that way a few months ago, but with the open bugs stalled I didn't bothe... [19:39:01] 6Labs, 6operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#1783491 (10scfc) (Ignore my last comment; I fell into the trap of not reloading the task page before clicking "Submit" again. Oh, Bugzilla, sometimes I miss you very much.) [19:39:07] (03CR) 10ArthurPSmith: "I uploaded a couple of changes to nuclides.py and the css file to display more granular coloring between 1 ms and 1000 s, and added some c" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 (owner: 10ArthurPSmith) [19:45:04] valhallasw`cloud, when you say the button submits for you, do you just mean it sends the request, or that you get the page saved? I'm seeing the first, Niharika is seeing the second. [19:46:39] PROBLEM - Host tools-andrew-puppettest is DOWN: CRITICAL - Host Unreachable (10.68.21.109) [19:47:39] ^ weirder and weirder: her changing to "button" solved that too [20:13:02] 6Labs, 10Labs-Infrastructure, 6operations: deployment tracking of codfw labs test cluster - https://phabricator.wikimedia.org/T117097#1783632 (10chasemp) I spoke with @papaul since he is going on vacation as it would be good to get the hosts mentioned [[ https://phabricator.wikimedia.org/T117107#1770439 | he... [20:15:43] Hello Everybody. I have a script, (anniversary.py) what is running excellent submitting with jsub, but queued for ever if I scheduled with my other scripts in crontab? Can anybody help me? [20:23:45] 10Tool-Labs-tools-Other, 6Community-Tech, 7Easy: Edit Count tool on labs shows text in wrong language - https://phabricator.wikimedia.org/T111512#1783675 (10kaldari) [20:28:51] fhocutt: if type='submit', it saves the page for me [20:29:26] might be another chrome/firefox thing? [20:30:47] ato_: which tool is this? [20:31:06] ato_: there should not be a difference between cron and shell for jsub :/ [20:32:45] i know it should not be differ but for me it is so [20:32:53] ato tool :) [20:34:58] ato_: so qstat -j shows you the scheduling information. In this case, qstat -j 1216034 [20:35:07] and it fails to schedule because it can't find a host with enough information [20:35:33] because you request 512 GB instead of the 512 MB you probably wanted to request :-) [20:39:45] Thank you qstat -j 1216034 say some memory problem... [20:41:09] valhallasw`cloud: stop being mean [20:41:14] if he wants 512GB... [20:41:36] pfff [20:41:52] my god [20:42:04] thanks [20:42:13] i try it again.. [20:42:32] ato_: you're welcome... these kinds of typo's are hard to debug :( [20:42:46] ato_: don't forget to stop the existing job -- qdel 1216034 [20:46:44] thanks again [20:48:10] working :D [20:48:16] cool! [20:48:36] good night :))) [20:48:57] 'night :-) [21:39:14] YuviPanda: in the short run what kind of queries do you need to run as ‘novaobserver’? More than just a list of all projects? [22:19:28] andrewbogott: list instances and their properties [22:19:33] specifically, ips [22:19:39] YuviPanda: ok [22:19:47] andrewbogott: I'm just going to merge that now [22:19:56] andrewbogott: I'm also ready to help with 1203 depooling ify ou want [22:20:02] merge which? [22:22:33] andrewbogott: the patch that starts querying ips and instances :) [22:22:37] oh [22:22:45] https://gerrit.wikimedia.org/r/#/c/243357/ [22:22:58] wait, I’ve been ignoring that patch since it does things the wrong way but I didn’t want to veto it... [22:23:04] does it use the wikitech api? [22:23:40] andrewbogott: no [22:23:42] andrewbogott: it hits nova [22:23:49] how? [22:23:57] oh, it runs on a production box [22:24:01] so not dangerous [22:24:01] andrewbogott: yeah [22:24:03] no [22:24:09] it'll run on holmium I guess [22:24:33] I tested the puppet parts on a labs box and the script itself on silver [22:28:44] andrewbogott: any other objections? [22:28:55] it'll currently hit the API once every 20mins though. [22:28:59] I’ll add comments, hang on... [22:29:09] Mostly I don’t like that it it’s async [22:29:16] but, like I said, I don’t want to veto [22:33:55] andrewbogott: what do you mean by async? [22:34:01] andrewbogott: oh that it isn't synchronous with the actual change? [22:34:18] yeah but I don't know if we have any 'notification' mechanism whenever anyone modifies IPs [22:34:24] currently they're all hardcoded in puppet tho :( [22:34:24] yeah — really it should be hooked when the dns record is changed. [22:34:38] yeah [22:34:52] but, it’s ok — this solution is probably better than the status quo [22:35:03] yeah, I guess eventually we should have designate do something [22:35:08] but that seems a fair bit more work [22:35:41] andrewbogott: ah require_package [22:35:49] andrewbogott: because I think designate runs on the same host? [22:35:56] and with that if I specify package for one or both they'll conflict [22:36:03] and won't with require_package [22:36:18] and the include changes to require because I can't do a require => Class[] for require_packages [22:37:00] that change can be in a preliminary cleanup patch though, can’t it? [22:37:50] I can split that out if you want, sure! [22:37:59] but will have no effect until this one goes in [22:38:44] I think I’d like it in a different patch, if only because that patch is confusing enough as it is :) [22:39:05] kk let me split it out [22:39:28] thanks [22:58:25] PROBLEM - Puppet failure on tools-exec-1204 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:02:08] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [0.0] [23:02:30] PROBLEM - Puppet failure on tools-packages is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:07:55] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:13] uh [23:08:21] wfm shinken-wm [23:08:24] that's all me probably [23:10:00] YuviPanda: the next patch i want to merge may or may not break the nova api. So probably you should finish what you’re doing first… let me know when you’re clear :) [23:10:18] andrewbogott: ok! [23:12:47] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 920976 bytes in 3.131 second response time [23:18:07] 6Labs, 10Labs-Infrastructure, 5Patch-For-Review, 3labs-sprint-119: Automate generation of floating/private dns aliases in the labs recursor - https://phabricator.wikimedia.org/T100990#1784096 (10yuvipanda) 5Open>3Resolved Bammm! All done now. [23:20:34] (03CR) 10Yuvipanda: [C: 032 V: 032] Don't allow creation of invalid domains [labs/invisible-unicorn] - 10https://gerrit.wikimedia.org/r/245200 (https://phabricator.wikimedia.org/T69927) (owner: 10Alex Monk) [23:27:23] 10Tool-Labs-tools-Other, 7I18n: [[Intuition:Raun-more entries/ksh]] i18n issue - https://phabricator.wikimedia.org/T112467#1784134 (10Krinkle) 5Open>3Invalid a:3Krinkle Per translatewiki.net conventions, report issues about messages at . As for... [23:30:42] 6Labs, 10Labs-Infrastructure, 5Patch-For-Review, 3labs-sprint-118: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1784146 (10yuvipanda) Needs package to be built and deployed still. [23:37:08] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [23:37:28] RECOVERY - Puppet failure on tools-packages is OK: OK: Less than 1.00% above the threshold [0.0] [23:38:26] RECOVERY - Puppet failure on tools-exec-1204 is OK: OK: Less than 1.00% above the threshold [0.0] [23:40:00] 10Quarry: Add 'download in HTML format' option (Quarry) - https://phabricator.wikimedia.org/T117644#1784169 (10yuvipanda) I suppose in addition to the CSV, TSV, JSON and JSON-Lines format the reporter would like 'HTML' as a download format as the results as well. [23:40:37] 10Quarry: Add 'download as wikitable' option (Quarry) - https://phabricator.wikimedia.org/T117645#1784170 (10yuvipanda) I suppose in addition to the CSV, TSV, JSON and JSON-Lines format the reporter would like 'wikitable' as a download format as the results as well.