[08:57:28] YuviPanda: this is for you http://ultimategerardm.blogspot.nl/2014/12/wikidata-kcg-college-of-technology.html [09:03:28] tto@tools-login:~$ sql labswiki_p [09:03:29] This is unknown db to me, if you don't like that, blame petan on freenode [09:03:53] I'm pretty sure this is tracked in phabricator, but I can't find it right now...? [14:14:56] there is an issue with receiving Openstreetmap maps [14:15:05] is this something that I should mention here ? [17:25:40] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:26:05] can someone please restart https://wikitech.wikimedia.org/wiki/User:Yuvipanda/Restarting_magnus_wdq [18:34:18] anyone with enough power around [18:37:58] I am so pissed of by the lack of service [18:45:04] aude: Can you restart wdq for GerardM- ? https://wikitech.wikimedia.org/wiki/User:Yuvipanda/Restarting_magnus_wdq [18:45:34] rage [18:45:38] i think it's on toollabs [18:45:58] * aude see if i have permission [18:47:42] aude you have [18:47:48] ok [18:48:08] I was told so [18:49:30] try again :) [18:51:10] I do not know what it all starts .... if it is MariaDB it takes its sweet time [18:51:37] i have no idea [18:51:51] :) that is ok [18:52:08] at least it's documented [18:57:24] aude: I've offered to magnus to help puppetize it and set up load balancing, etc [18:58:59] YuviPanda: would be great [19:00:56] aude: yup but I need to understand what the system does a bit more before I can do that. Waiting to hear back from.magnus again [19:01:41] * aude nods [19:03:42] aude: I hear there is a project to build something similar for prod [19:04:38] yeah, for wikigrok etc [19:04:51] they use wdq now afaik [19:09:37] aude: yeah. I don't think it is at a scale to affect wdq stability tho [19:48:40] YuviPanda: is there a quarry test instance I can fiddle with? [19:48:47] I'd like to add some metadata to the json output [19:48:54] valhallasw`cloud: oooh, like what? [19:49:06] author, title, description, SQL, status [19:49:10] basically :-p [19:49:23] valhallasw`cloud: hmm, isn't that already there? [19:49:31] data = { [19:49:31] 'headers': header, [19:49:31] 'rows': rows [19:49:31] } [19:49:41] valhallasw`cloud: I mean, not in the JSON output files, but there's other API calls for it [19:49:52] valhallasw`cloud: but sure, if you want to add it to the output files, fine. [19:50:02] YuviPanda: i can haz docs? :-p [19:50:07] valhallasw`cloud: I'm ok with just testing on the main instance itself :P but if you want it's trivial to create another testing instance. [19:50:11] since everything is puppetizes [19:50:14] *puppetized [19:50:22] valhallasw`cloud: for the code? aaaah, see bug #1, sadly [19:50:31] but I'll help with whatever, and these should be trivial to add [19:50:31] I think you mean T2001 ;-D [19:50:35] haha :) [19:50:42] but, analytics/quarry/web [19:50:47] and it really should be trivial to add [19:52:02] I don't think those are in the api, actually [19:52:23] there's /api/query/meta [19:52:41] but that actually sets metadata [19:53:32] hmm, right [19:53:37] valhallasw`cloud: you can only get the status from the API, I guess [19:54:27] YuviPanda: could you add me to the quarry project, then? I'll try to spin up a new testing instance. [19:54:39] (is that what quarry-web-test was supposed to be?) [19:56:34] also, is it by design non-published queries are shown in the query list? [19:57:46] valhallasw`cloud: yes. everything is public, 'published' just means... more polished or something [19:57:50] ok [19:57:51] valhallasw`cloud: adding you momentarily [19:57:55] <3 [19:58:59] valhallasw`cloud: added [19:59:16] ok, let's see if I can not-break stuf [19:59:21] valhallasw`cloud: hmm, setting up a pure web instance is going to take some configuration. [19:59:29] mm. [19:59:37] valhallasw`cloud: wait, do you want to re-use the db and redis queue of the main setup? [19:59:42] valhallasw`cloud: or do you want a completely isolated setup? [19:59:43] sure [19:59:54] I just want to fiddle with the output without breaking everything :-p [20:00:00] hehe [20:00:07] then former, and would require some configuration then [20:00:24] valhallasw`cloud: can you create a small instance, quarry-web-test? I'll look at the puppet code in the meantime [20:00:49] YuviPanda: no ;-) you added me as user, not as admin ;-) [20:00:55] valhallasw`cloud: I did that as well [20:01:01] > Successfully added Merlijn van Deen to projectadmin. [20:01:04] oh, labs is just slow then I guess [20:02:33] I... have no clue how to do that. *opens docs* [20:03:00] valhallasw`cloud: wheee! :D Special:NovaInstance [20:03:41] YuviPanda: that's completely empty for me [20:03:44] valhallasw`cloud: welcome to non-tool-labs! [20:03:46] even if I select quarry in projects [20:03:49] valhallasw`cloud: log out and back in. wikitech bug [20:03:55] .... [20:03:59] >_< [20:04:01] inorite [20:04:36] yes, now there's a miniscule add instance button. Good. [20:04:41] You cannot complete the action requested as your user account is not in the project quarry. [20:04:47] wat. [20:04:47] ಠ_ಠ [20:04:52] did you log out and back in? [20:05:08] yep. let me do that again [20:05:18] valhallasw`cloud: lololol, I need to log out and back in >_> [20:05:25] ...wat [20:06:20] valhallasw`cloud: i mean, it was blank for me as well [20:06:24] ah [20:06:25] valhallasw`cloud: but yeah, I see you as projectadmin [20:07:26] also, WTF does search not search in the nova project namespace [20:07:44] welcome to wikitech, etch [20:07:44] *etc [20:08:11] https://phabricator.wikimedia.org/T67132 [20:08:29] anyway, https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry does not list me as admin [20:09:12] valhallasw`cloud: just removed and re-added you [20:09:40] UGGGH and I can't find anything anymore in the tool labs docs with everything split up and in popout stuff [20:09:48] inorite... [20:09:52] I hate change ;-p [20:10:28] so say we all :) [20:17:17] anyway, still can't view https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=quarry&instanceid=46002ce4-49b7-46eb-943e-09ed2bd7d341®ion=eqiad for instance [20:20:47] valhallasw`cloud: hmm, blah. [20:20:49] valhallasw`cloud: can you ssh in? [20:20:55] probably :-p [20:21:07] oh I have an idea [20:21:10] I can just take the same instance [20:21:17] but spin one up by running app.py directly [20:21:37] valhallasw`cloud: wooo, yes, you can do that. [20:21:48] bastion doesn't have mosh -_-' [20:21:48] valhallasw`cloud: that would also mean you can re-use the existing config there, and all shall be well [20:21:52] nope [20:21:54] there's a bug for it [20:22:00] mosh doesn't do proxycommand nor agent forwarding [20:22:02] can't use [20:22:03] sadly [20:22:15] *shrug* I just have a labs private key in my ~/.ssh :P [20:23:20] valhallasw`cloud: yeah, I'm going to setup a script sometime to catch people doing that and disable all those keys... [20:23:21] :P [20:23:33] YuviPanda: because... why exactly? [20:23:50] people with root can abuse agent forwarding as well [20:24:11] valhallasw`cloud: indeed, and it would be nice to disable *that* as well :P [20:24:16] valhallasw`cloud: but that's probably not going to happen [20:24:19] so fix host-based auth [20:24:25] on where? [20:24:31] on ALL the servers :-p [20:24:34] basically, bastion is trusted [20:24:39] hmm, actually [20:24:41] that's not that bad an idea. [20:24:50] I wonder why hadn't thought of that yet. [20:24:53] let me file a phab ticket [20:25:36] okay how do I even log in... ssh to quarry.wmfglabs.org fails, but that's probably because that's a proxy. bastion.wfmlabs.org is unreachable from tools? sigh. [20:25:48] ah, just ssh quarry-main-01 [20:26:42] yeah, that works [20:27:04] valhallasw`cloud: https://phabricator.wikimedia.org/T76971 [20:27:14] valhallasw`cloud: you should find it in /srv/quarry [20:27:36] ...instead of /data/project/quarry, /home/quarry or any of the other quarries :P [20:30:03] valhallasw`cloud: the former are on NFS. performance, etc :P [20:33:28] ugh and I need to tunnel further with the internal SSH? [20:33:29] ffs [20:34:22] okay that works [20:42:51] valhallasw`cloud: I'm trying to use your flask-mwoauth. [20:43:11] a930913: what's broken this time ;-) [20:43:17] valhallasw`cloud: :D [20:43:24] So it seems to log in. [20:43:37] But then it reports the user as None. [20:43:41] So it hasn't :p [20:44:27] a930913: mmm [20:44:29] YuviPanda: As for mwoauth, I have the access_token, but can't seem to use it to edit. [20:44:49] a930913: is it on TL? [20:44:54] or are you testing locally? [20:45:12] valhallasw`cloud: TL=tools-login? [20:45:18] tool labs [20:45:41] Ah, more specific than just labs I guess :p [20:45:44] Yes, it is. [20:45:54] oar/oauth [20:46:44] valhallasw`cloud: Also a bit or mod_rewrite. [20:47:00] valhallasw`cloud: Using fastcgi. [20:47:25] URLLLLLLLLLLLLLL :P [20:47:57] (legoktm said you might have a better way to test than repeated webservice start/stops) [20:48:22] valhallasw`cloud: http://tools.wmflabs.org/oar/oauth/ http://tools.wmflabs.org/oar/oauth/login [20:49:24] a930913: *nod* just use python app.py and adapt the return url manually [20:49:49] oh I'm of course not allowed to test [20:49:52] ugh. [20:51:45] a930913: lighttpd is basically a PITA to debug because you can't debug [20:52:39] valhallasw`cloud: Not allowed to test? Add to service group? [20:52:47] a930913: nah, has to do with oauth [20:53:12] valhallasw`cloud: You can't use your account with oauth? [20:54:03] until the app is approved, only the requester can use it [20:55:04] Oh. [20:55:09] That was explained nowhere. [20:55:24] I thought it suspicious that it could be used without approval. [20:55:53] Erm, who can approve? Should just be a matter of rubber stamping, no? [20:56:27] https://www.irccloud.com/pastebin/oNYdjcql [20:56:29] YuviPanda: ^ [20:56:50] valhallasw`cloud: is this for a meta query? [20:56:57] YuviPanda: no, just added to the .json output [20:57:08] so meta is next to headers and rows [20:57:14] valhallasw`cloud: oh, then include query id, query rev id, query run id, and query run timestamp? [20:57:39] YuviPanda: mmm [20:58:45] YuviPanda: ... [20:58:49] YuviPanda: which one is which?! [20:58:57] query.id is not the id in the url [20:59:10] oh, that's becaue that's the run id [20:59:11] I see [20:59:47] valhallasw`cloud: What should I do to debug it? [20:59:58] a930913: as mentioned, run it using app.py and try manually [20:59:59] valhallasw`cloud: right, so query id is like pageid, query rev id is like revid, and query run id is a particular revision of the query run at a particular time producing a particular output [21:00:30] YuviPanda: right [21:00:46] valhallasw`cloud: It just returns / and exits. [21:00:53] a930913: ??? [21:01:24] Oh it's the WSGI stuff I think. [21:01:39] app.py, not app.fcgi [21:01:42] man, I really need to get uwsgi webgrid nodes running, don't I [21:01:58] valhallasw`cloud: What's the difference? :/ [21:02:12] a930913: one is the fcgi wrapper, the other is the actual app [21:02:26] Oh, I have it all the same :p [21:02:58] ...wat. [21:03:08] don't do that :P [21:05:36] valhallasw`cloud: Ok, so got app.py running the flask. Do I use elinks in another terminal or something? [21:07:26] YuviPanda: what's extra_info for? [21:07:33] queryryn.extra_info [21:07:53] a930913: preferrably do everything locally, but that's an option yes [21:08:03] a930913: I don't really have time to walk you through debugging right now though [21:08:03] valhallasw`cloud: stores json-blob of arbitrary info that isn't queried. currently, I think, number of resultsets and number of rows in each [21:08:26] and task_id = string? [21:09:14] valhallasw`cloud: task_id is the *celery* task id [21:09:18] achso [21:09:19] valhallasw`cloud: used to cancel tasks and check their status [21:16:47] TypeError: datetime.datetime(2014, 12, 7, 20, 48, 38) is not JSON serializable [21:16:49] uggghhhh [21:18:13] valhallasw`cloud: inorite. that's so pissing off... [21:20:55] https://www.irccloud.com/pastebin/4F4Z7xlL [21:20:58] ^ [21:20:59] YuviPanda: [21:22:13] YuviPanda: and I guess it's not currently possible to re-run a query, give nthat rev_id==run_id? :p [21:22:37] I might take the non-annotated SQL though [21:22:54] valhallasw`cloud: yeah, no UI for it :) [21:23:04] valhallasw`cloud: sooo... the output of the JSON has to be fully cacheable forever :) [21:23:17] ~_~ [21:23:24] valhallasw`cloud: so you should kill the latest_run properties [21:23:28] valhallasw`cloud: in fact, why dump them all? [21:23:36] valhallasw`cloud: why not just put query id, rev id, run id, and run timestamp? [21:23:58] ...okay, but then this exercise is utterly useless [21:24:05] why? [21:24:09] because hten I also cannot dump the query title and description [21:24:23] oh, damn. [21:24:43] which is fine, I can just make /query/123 return something else when json is asked for [21:24:46] hmm, that might have been the reason I didn't put them in in the first place :| [21:24:59] why does it need to be cacheable for infinity and beyond? [21:25:26] scalability reasons? Quarry was designed for random applications to be able to hit the JSON output endpoint via CORS [21:25:30] and just... use that. [21:25:34] to do things [21:25:51] plus retreiving results is a fairly biggish operation - hits NFS, reads from sqlite... [21:25:55] mm. [21:26:13] so, my suggestion would be to just put query id, rev id, run id in the output - those won't change [21:26:23] and then just lookup whatever you need with an additional API request [21:26:35] non-existing api request :P [21:26:44] that, yeah :) [21:26:51] but if you write this and don't want to write that, I can help ;) [21:27:04] or you could write both :D it's fairly trivial [21:28:09] nothing is ever trivial :P [21:29:02] it's all relative :) [21:33:30] YuviPanda: so how would I get sql for a revision...? [21:33:36] which endpoint? [21:34:32] valhallasw`cloud: /api/query-revision/? [21:34:45] should return timestamp, sql, query id and latest run id? [21:35:29] mmm maybe I should just make "/run//meta//json" ? [21:35:51] mm not sure [21:36:49] YuviPanda: wait. what's resultset_id within run_id?? [21:36:53] now there's four layers [21:36:58] query, rev, run, resulset [21:37:16] valhallasw`cloud: well, an SQL query can produce multiple resultsets. two select commands separated by ; for example [21:37:18] valhallasw`cloud: A bit of crafting URLs later and I now have "OAuthException: No token available" :/ [21:37:26] YuviPanda: oh. Right. [21:37:40] valhallasw`cloud: yeah, it was a painful re-architecture when I realized that :| [21:37:58] valhallasw`cloud: Oh, hold on. [21:38:03] where's the number of resultsets storef? [21:38:13] not in https://github.com/wikimedia/analytics-quarry-web/blob/master/quarry/web/models/queryrun.py [21:38:32] valhallasw`cloud: extra_info [21:38:41] oh, it's majick. [21:38:47] as I said, it *was* bolted on... [21:38:54] valhallasw`cloud: extra_info is json [21:38:55] :-p [21:39:03] valhallasw`cloud: you can look at the db itself. [21:39:11] but be careful, I've no backups of that db :P [21:39:13] (need to fix) [21:39:47] Nope, still throwing that error :( [21:39:54] valhallasw`cloud: also, what's your general use case? what're you building this out for? [21:40:06] YuviPanda: in this case, a tool to convert it to a tsreports report [21:40:16] oh, interesting. [21:40:18] why? [21:40:21] better formatting, etc? [21:40:31] yes, and better caching, and easier cross-site usage [21:40:34] and recurrance? [21:40:39] what do you mean by 'cross site usage'? [21:40:58] running it on other sites for non-techies :P [21:40:59] aah, I see [21:41:01] right [21:41:05] yeah, that makes sense. [21:41:19] I'm going to spend a few days on quarry this week, but that's going to be code cleanup. [21:41:34] and then user groups, so some users can run longer queries (for an hour, maybe) [21:41:37] I'm also fine with queries that take a week; everything is cached to {runtime x 100} or so [21:41:49] well maybe not a week but a day or a few hours :P [21:42:08] which you probably don't want for quarry [21:42:20] indeed. [21:46:45] valhallasw`cloud: API "The authorization headers in your request are not valid for mediawikiwiki" [21:47:04] a930913: yeah, so there's something wrong with the oauth handshake beforehand [21:47:29] valhallasw`cloud: Is that my problem or your problem? [21:47:36] I don't know. [21:48:21] I'll try poke around to see if it's my problem, otherwise I'll wait until you have more time ;) [21:51:00] Weird, I get one error on labs, and another error on local. [21:53:55] !screen | YuviPanda [21:53:55] YuviPanda: script /dev/null [21:54:10] YuviPanda: why is git-review not installed theeeeeeeere ;( [21:54:16] !screen | YuviPanda [21:54:16] YuviPanda: script /dev/null [21:54:24] valhallasw`cloud: I don't use git-review :P [21:54:24] why is git-review not in the fscking standard puppet setup [21:54:29] FINE. [21:54:29] :p [21:54:39] because it's annoying [21:57:18] not more than the rest of gerrit :P [21:57:32] that's true [21:57:35] but removing pain [21:57:36] is [21:57:39] niccceee [21:58:30] remote: ERROR: missing Change-Id in commit message footer [21:58:33] FFFFFFUUUUUUUUUUUUUUUUUUUUUU [21:58:39] how is that better than git-review :P [21:59:47] and git-rebase IS BROKEN [21:59:50] GRRRR [22:00:39] YuviPanda: if i need some packages on tools, phab request ? [22:00:57] valhallasw`cloud: you can just install git-review :D you've rot [22:01:00] roooot [22:01:03] matanya: on toollabs or labs in general? [22:01:13] toollabs [22:01:20] on laabs in general i'm a sudoer [22:01:23] YuviPanda: LOL. Right :D [22:01:32] matanya: yes, phab request :) [22:01:39] thanks [22:01:45] matanya: and just point me to the link, I'm sure I can get them done shortly [22:01:52] thanks! [22:04:13] YuviPanda: finally :P https://gerrit.wikimedia.org/r/#/c/178128/ and https://gerrit.wikimedia.org/r/#/c/178129/ [22:04:40] flake8 [22:04:44] okay I'm done for today :P [22:05:43] valhallasw`cloud: :D thank you! [22:05:48] * YuviPanda hugs valhallasw`cloud [22:06:11] YuviPanda: https://phabricator.wikimedia.org/T76974 [22:06:25] I might find out i need more after it, if it is ok with you [22:06:37] matanya: yeah, sure! [22:11:27] YuviPanda: right, as_json. Vous avez raison. [22:12:07] valhallasw`cloud: :D I understood that french, I think [22:12:11] 'you speak reason'? [22:14:11] matanya: patch merged, am forcing a run on -login [22:14:28] "you have a point" [22:14:38] anywayz, bed [22:14:43] valhallasw`cloud: damn. [22:14:45] valhallasw`cloud: night! [22:15:29] matanya: goddamit, someone broke puppet on all of labs earlier. [22:31:36] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:32:08] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [22:34:41] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [22:34:43] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [22:35:03] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [22:38:25] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [22:38:29] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:38:41] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [22:40:07] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:40:20] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [22:40:32] RECOVERY - Puppet failure on tools-trusty is OK: OK: Less than 1.00% above the threshold [0.0] [22:40:36] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [22:40:40] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:20] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:56] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:57] legoktm: btw, the shinken warnings you got was just a labs wide puppet failure, I fixed [22:42:20] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [22:42:28] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [22:43:52] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:24] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:56] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:58] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [22:46:22] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [22:46:28] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [22:46:48] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:46:50] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:47:24] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:48:31] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [22:48:31] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [22:50:17] When I try to connect from one of my tools to another one through a XHR, I get a "403 Forbidden" error by nginx/1.5.0 -- It works fine with wget ... what's the reason [22:50:23] I am using node.js [22:50:48] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:50:54] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:51:25] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [22:51:32] YuviPanda: need help with puppet on labs ? [22:51:55] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [22:52:07] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:52:17] rillke: heya! set a user agent :) [22:52:21] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [22:52:41] matanya: nah, it’s ok now, I just unbroke it. It was broken yesterday with the base lint change that was trivial, but accidentally removed an include [22:52:46] matanya: thanks for asking! [22:53:14] the lint by alex? [22:53:23] matanya: yeah. [22:53:29] matanya: I fixed it tho [22:53:29] saw that one [22:53:35] meh [22:53:36] ok [22:59:05] YuviPanda, you are the best :) This has been bugging me for a year now :P [22:59:24] rillke: :D [23:00:03] rillke: I suppose I should send back a body with the 403 saying ‘you need to supply a user agent' [23:03:22] rillke: we put this in place on tools only a few months ago tho. been in on prod forever but [23:04:57]