[00:22:51] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:09:06] yuvipanda: is there an easy way that I search over the queries in the recent or published subsets in quarry? [04:35:21] Read: Mark Dice [04:35:21] Big Brother: The Orwellian Nightmare Come True [06:54:57] PROBLEM - Puppet run on tools-exec-1215 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:34:57] RECOVERY - Puppet run on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [08:09:30] PROBLEM - Puppet staleness on tools-checker-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [08:17:02] Hi! I got very often this error last hour, what's going on: "Request from 10.68.23.58 via cp1054 cp1054, Varnish XID 3034264233
Error: 503, Service Unavailable at Fri, 23 Sep 2016 08:15:12 GMT" [08:19:17] yuvipanda: is it possible to say something to this? ^ [08:25:08] doctaxon: uh, more details please. what URLs are you seeing that on? [08:31:13] doctaxon: in general, if you are seeing repeated 503s for simple API queries like that, yes please file a bug [08:34:20] doctaxon_: [01:31:13] doctaxon: in general, if you are seeing repeated 503s for simple API queries like that, yes please file a bug [08:34:38] okay [08:36:42] legoktm: but can we do something in this minutes too? [08:36:58] well, I don't really know what's wrong [08:37:06] 503 is typically something that ops looks into [08:42:00] legoktm: what is a cp1065 [08:42:02] (03PS105) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [08:42:24] doctaxon: caching proxy, a server that runs varnish [08:42:55] (03CR) 10jenkins-bot: [V: 04-1] Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [08:43:23] (03CR) 10Legoktm: "Eek, PS105??? What's blockning getting this merged? Let's not let perfect be the enemy of the good here..." [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [08:44:44] 06Labs, 10Tool-Labs, 06Operations: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661551 (10doctaxon) [08:45:27] 06Labs, 10Tool-Labs, 06Operations: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661564 (10doctaxon) p:05Triage>03High [08:47:33] (03CR) 10Ricordisamoa: "> Eek, PS105??? What's blockning getting this merged? Let's not let" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [08:49:38] (03CR) 10Ricordisamoa: "PS105 updates Bootstrap/Bootswatch from 3.3.6 to 3.3.7, jQuery from 2.2.3 to 3.1.0" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [08:58:21] (03PS106) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [09:02:39] (03CR) 10Ricordisamoa: "PS106 updates grunt-stylelint from ~0.3.0 to ~0.6.0, stylelint-config-wikimedia from ~0.2.0 to ~0.3.0" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [09:21:44] (03PS21) 10Ricordisamoa: Initial commit [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 [09:23:52] (03CR) 10Ricordisamoa: "PS21 updates Bootstrap from 3.3.6 to 3.3.7, jQuery from 2.2.0 to 3.1.0" [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 (owner: 10Ricordisamoa) [10:01:18] Who could I ask to take down a specific wmflabs instance (hosting the raw DevSummit2016 videos, which isn't needed anymore as they are now on Commons)? The original owner told me they currently don't have time or a good connection to do it themselves... [10:01:19] TIA [10:29:26] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:31:17] andre__ create a ticket for #labs-infrastructure , people there will be glad to save space and resources :-) [10:32:33] Makes sense. Thank you! [10:34:03] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661922 (10jcrespo) Adding Traffic so they can give it a quick look. [10:37:51] 06Labs, 10Labs-Infrastructure: Switch off a specific wmflabs instance - https://phabricator.wikimedia.org/T146466#2661925 (10Aklapper) [10:40:11] andre__, do you know the name of the instance, that would help making it faster [10:40:47] (I am not in charge of that or have permissions, just a suggestion) [10:41:20] I know the public URL, yeah. But already filed a ticket. It's privacy related (but not urgent) hence I'm a bit vague intentionally [10:41:30] ok [10:41:35] understood [10:41:59] thanks for the help! :) [10:59:27] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:19:47] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661961 (10doctaxon) p:05High>03Unbreak! changed Priority because there have to run a lot of bot scripts Wikipedia users needs to work with it. The unbreak is open.... [11:26:34] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661965 (10jcrespo) @doctaxon can you indicate the full url you are trying? [11:29:00] jynus: full url? I am trying it by API [11:29:27] format json / maxlag 5 / action query / prop info / titles {:Kurt Couto} [11:29:38] yes, but mediawiki api requires a host an an url, which one are you using? [11:30:37] e.g. https://en.wikipedia.org/w/api.php?action=my_call [11:31:54] maybe you are not using https, which is required [11:31:55] this is an example of many: https://de.wikipedia.org/w/index.php?title=Kurt_Couto&action=info [11:32:03] ok [11:32:20] thank you, can you add that to the ticket so more people see it? [11:32:33] if we have the url, we can see the logs, so it is important [11:32:49] doctaxon: antwortet dir das api? (habe jetzt nicht mitgelesen), ich kriege keine aw fom api [11:34:24] Steinsplitter: see error line in T146451 [11:34:24] T146451: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451 [11:35:02] this is what I got in the shell, on bastion and the grid [11:35:40] same problem here. [11:35:50] jynus: I got the error now from cp1053 [11:37:27] could you both please add the url you are trying on the ticket, it is more information to investigate [11:37:54] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661968 (10doctaxon) from chat about the topic: ``` 11:26 < wikibugs> Labs, Tool-Labs, Operations, Traffic: repeated 503 errors for 90 minutes now on... [11:39:56] jynus: it's on the ticket now [11:41:39] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661971 (10Steinsplitter) Getting the problem when accessing the Wikimedia Commons api via labs or labs grid engine. For example when attempting to getting image info... [11:43:22] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661551 (10Joe) @Steinsplitter do you get the data correctly if you try from your computer? [11:43:33] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661974 (10doctaxon) is the error related to the cache proxies, if there are reports of all the cp1065, cp 1053, cp 1055 ...? [11:44:42] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661975 (10Joe) >>! In T146451#2661974, @doctaxon wrote: > is the error related to the cache proxies, if there are reports of all the cp1065, cp 1053, cp 1055 ...? It... [11:47:33] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661977 (10doctaxon) Who is responsible for that? [11:52:40] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2661984 (10Steinsplitter) >>! In T146451#2661972, @Joe wrote: > @Steinsplitter do you get the data correctly if you try from your computer? Yes, ~ 40 successful attem... [12:04:14] so this is the status, we believe to have find the cause, we are trying to fix it [12:05:37] please provide feedback when things get getter [12:06:45] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662029 (10doctaxon) next error trying this: https://de.wikipedia.org/w/index.php?title=Offshore-Windpark_Borssele&action=info ``` / format json / maxlag 5 / action q... [12:08:16] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662035 (10Joe) @doctaxon do you get an error consistently for that url? if so, trying from where? I still can't reproduce your problem, that seems not to be limited t... [12:09:18] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662036 (10doctaxon) no, it's not consistently but random, it's always API info up to now [12:18:22] when opening in browser format=json the content is rawtext but when wget'ing it ... it is html. Maybe a silly question, but is that normal? [12:26:42] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662047 (10doctaxon) runs good for 8 minutes now [12:28:27] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662048 (10ema) I've tried reproducing the issue for a while without success. @Joe restarted mw1280-90 due to memory leaks, perhaps that helped? [12:51:18] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662098 (10doctaxon) Okay, I suppose, the problem has been solved. What have you done to solve it? [12:53:46] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662101 (10ema) @doctaxon: nothing, except for @Joe's restart of the HHVMs mentioned above. [12:56:05] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662109 (10Joe) @doctaxon I tracked down `mw1203` and `mw1280-1290` as potential source of problems because of how much cpu/RAM they were consuming, and issued a rollin... [12:58:20] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662110 (10doctaxon) Top! Thank you very much! [13:05:34] 06Labs, 10Tool-Labs, 06Operations, 10Traffic: repeated 503 errors for 90 minutes now on cp1065 - https://phabricator.wikimedia.org/T146451#2662114 (10Joe) 05Open>03Resolved a:03Joe [13:29:41] doctaxon, "headers(MediaWiki-API-Error) = readonly" means that you have to wait [13:31:18] jynus: but a script does not wait [13:31:35] well, teach it :-) [13:31:39] yeah [13:31:51] I can explain why that happens [13:31:55] if it is useful [13:32:32] and why is not an error, just a protection [13:32:48] explain please [13:33:06] so, when the servers are either too loaded, or there is maintenance [13:33:16] they go into read-only mode [13:33:39] if editing was allowed, you could be reading an old version of the pages and editing such a version [13:34:10] to prevent that, mediawiki says that it is in read only mode and refuses edits [13:34:18] that is not a normal state [13:34:33] just wait, and in a few seconds you should be able to edit again [13:34:56] if that wasn't there, bots couls revert edits without noticing them [13:35:06] okay thank you [13:35:19] there were 2 spikes recently [13:35:25] but things should be ok now [15:18:50] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:39:55] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Move designate and nova plugins out of packages and straight into the puppet repo - https://phabricator.wikimedia.org/T144317#2662577 (10AlexMonk-WMF) 05Open>03Resolved a:05Andrew>03AlexMonk-WMF This is done now, there's just one remaining commit to... [15:40:30] 10Tool-Labs-tools-Wikidata-Periodic-Table, 10Wikidata: ptable app is broken again! - https://phabricator.wikimedia.org/T142432#2662580 (10ArthurPSmith) Excellent, thanks! I probably should have sent you an email... [15:44:44] That feeling when you find a perfect python package for a task you are working on and then discover that it has 10+ open PRs and no new releases in 2+ years [15:53:20] 10Quarry: json-lines output format doesn't have line breaks - https://phabricator.wikimedia.org/T146482#2662609 (10Halfak) [15:53:52] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:54:06] 10Quarry: json-lines output format doesn't have line breaks - https://phabricator.wikimedia.org/T146482#2662622 (10Halfak) [15:54:29] 10Quarry: json-lines output format doesn't have line breaks - https://phabricator.wikimedia.org/T146482#2662609 (10Halfak) [15:54:41] 10Quarry: json-lines output format doesn't have line breaks - https://phabricator.wikimedia.org/T146482#2662609 (10Halfak) [16:15:35] 10Quarry: Add SHOW EXPLAIN support to Quarry - https://phabricator.wikimedia.org/T146483#2662660 (10bd808) [16:36:39] o/ yuvipanda [16:36:46] Any chance I can get you to look at https://gerrit.wikimedia.org/r/#/c/312531/ quick? [16:37:04] I think it'll be a mega-easy review [16:57:05] halfak: we're in code-freeze so it probably won't be merged for a bit (unless you convince Yuvi that it's an emergency.) [16:57:33] andrewbogott, code freeze for quarry? [16:57:54] code freeze for the whole Ops team is at an offsite until the 1st. [16:58:04] Oh!!! [16:58:18] Boo. This is a blocker for me, but I guess it isn't an emergency [16:58:50] sed -r "s/}/}\n/" is a fine work-around. [16:59:01] *sed -r "s/}/}\n/g" [16:59:21] Since the JSON blobs can't have sub-objects [16:59:34] Hopefully none of the strings have a "}" in them :/ [17:09:43] halfak: that looks pretty trivial. who else admins quarry? [17:09:57] No one that I know of other than yuvipanda [17:10:09] that's ... not optimal [17:10:14] Agreed. [17:10:27] I'd be willing to take on a role like that. [17:10:36] I'll have to talk to yuvipanda about that [17:13:43] the whole ops team? I got stuff reviewed earlier.. [17:15:08] yup, we're in freeze now for a week [18:07:40] halfak: I don't think the freeze applies to quarry [18:08:08] halfak: bd808 indeed, I'd love to have people not me help with Quarry :) [18:08:40] yuvipanda, maybe it would be a good idea to train me in on deploys and your expectations. :) [18:08:50] +1 [18:08:53] (at some point when you aren't offsite) [18:09:10] not offsite yet, but won't have time to do that today (train you) but will do later! [18:09:16] I also want to move quarry to tools at some point to make this easier [18:10:20] halfak: am deploying your change now [18:10:27] \o. [18:10:29] \o/ [18:10:39] halfak: try now [18:11:15] Looks great [18:11:19] Thanks yuvipanda :) [18:11:20] <3 [18:11:33] halfak: awesome. thanks for the patch :) [18:22:44] halfak: I'll co-ordinate to schedule time with you maybe in 2-3 weeks to go over quarry [18:22:58] Sounds good. :) [18:23:33] ok [18:23:51] :) [18:38:05] 10Quarry, 13Patch-For-Review: json-lines output format doesn't have line breaks - https://phabricator.wikimedia.org/T146482#2663127 (10Halfak) Works now! Thank you. :) [18:38:17] 10Quarry, 13Patch-For-Review: json-lines output format doesn't have line breaks - https://phabricator.wikimedia.org/T146482#2663129 (10Halfak) 05Open>03Resolved a:03Halfak [19:36:58] 10Tool-Labs-tools-Pageviews: Add 'offset' or pagination to Massviews - https://phabricator.wikimedia.org/T144760#2663228 (10MusikAnimal) 05Open>03declined Maybe we'll add this... but I feel like with the new AQS cluster in place we can get away with processing the entire dataset in one succession. Right now... [19:37:18] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:36:27] Hey yuvipanda. I have a query "run" ID. How do I find the query from which it originated? [20:36:33] (Re. Quarry) [20:36:57] what are you trying to do, as such? [20:37:16] there's no way right now to map that in a public way (query run id to query) [20:37:21] can look up on thedb [20:37:59] Darn. So I have links to result downloads in my revscoring makefiles. [20:38:12] e.g. wget -qO- http://quarry.wmflabs.org/run/42221/output/0/json-lines?download=true > datasets/eswiki.sampled_revisions.20k_2015.json [20:38:22] if you have a query id [20:38:24] But I want to go back and look at some of the queries. [20:38:29] there's a way to get 'latest run of this query' from that [20:38:36] No query id [20:38:39] I can lookup the query for you [20:39:04] * halfak gets the id he wants [20:39:25] https://quarry.wmflabs.org/run/59580 ... [20:39:26] That one [20:39:30] halfak: sure, let me know [20:39:36] halfak: I'm also going to just add you to the project [20:39:40] \o/ [20:39:59] I think we need a word for this kind of volunteer pattern. [20:40:13] Procrastination is where you screw over future you by not doing what you are supposed to do [20:40:32] is where you screw over future you by signing up for some work that you don't have time for [20:40:38] heh [20:40:49] == "labs" [20:40:54] lol [20:41:00] halfak: what's your wikitech username? [20:41:07] 'Halfak' [20:41:56] !log quarry add halfak as projectadmin [20:41:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL, Master [20:42:13] halfak: I can lookup the query id for you, you can also dig into the mysql directly if you wish [20:42:21] (it is on quarry-main-01) [20:42:39] yuvipanda, I'll get it. Thanks for your help [20:42:49] halfak: \o/ awesome. [20:44:19] Awesome. Got tit [20:44:21] *it [20:44:37] https://quarry.wmflabs.org/query/7046 [20:44:51] \o/ cool [20:49:42] !log ores restarted ores-uwsgi and precached services on ores-web-03 [20:49:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [20:55:58] (03PS1) 10Krinkle: Fix broken Special:AbuseLog link (add missing '?' before query string) [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/312601 [20:56:16] (03CR) 10Krinkle: [C: 032 V: 032] Fix broken Special:AbuseLog link (add missing '?' before query string) [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/312601 (owner: 10Krinkle) [20:58:25] (03Merged) 10jenkins-bot: Fix broken Special:AbuseLog link (add missing '?' before query string) [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/312601 (owner: 10Krinkle) [22:40:14] halfak: i !bash'ed your insight -- https://tools.wmflabs.org/bash/quip/AVdZNFlcaH8PnNb4EBDP [22:40:40] :D! [22:40:43] I'm honored [22:43:02] my replacement for is probably "puppet" [23:15:52] bd808: you're probably right [23:17:12] I save myself the drain of time that is puppet by using.. alternative configuration management systems [23:17:59] I just wander into refactoring our puppet code on occasion and lose hours [23:18:16] and mw-vagrant puppet has been a large investment for me in the past [23:21:25] There's the puppet failures that have to be dealt with too. [23:22:39] I like serverless puppet much better, but we have prod secrets that we need a puppetmaster or another secret distribution system for [23:34:41] I find it hard to get stuff into the labs puppetmaster because it needs all the code review [23:35:48] Puppet is good in that it works right from when the instance was created, rather than me having to go manually install agents and then deploy [23:36:18] Puppet would be useful to do stuff like that. Downside is I know no puppet. [23:37:00] tom29739: oh, yes. Getting new code into ops/puppet can be quite a challenge. I'm "lucky" that most of the things I need to mess with have production uses as well. [23:37:22] I also have learned who I can pester for merges [23:37:48] "someday" we will hit on a better system for project puppet [23:38:39] y.uvi took a shot at it quite a while ago now with a way to run a second puppet run inside the primary but that had a lot of problems [23:39:18] the labs-vagrant project had similar issues. Thins would go badly when the mw-vagrant and ops/puppet code both wanted to manage a file [23:39:27] A system where I can add my own puppet modules without having to go through code review and not having to run a project puppetmaster would be great. [23:39:40] That's a good point. [23:39:56] *nod* that's the grail we are looking for, but I don't think anyone is actively looking right now [23:40:26] I can't edit some files for instance, because they're overwritten by puppet. [23:40:39] One of the problems becomes that you need to have all the modules on the shared puppetmaster [23:41:04] and that has security risks because of how puppet works [23:41:05] ops/puppet is a git repo, so why not have multiple git repos on there? [23:41:34] one of the security problems is that any puppet module can inject arbitrary ruby into the puppetmaster [23:42:02] If someone has access to the puppetmaster, then they have access to all the instances. [23:42:09] which is obviously a potential problem: arbitrary code running as root on a shared host [23:42:39] I use self-hosted puppetmasters in lots of projects to work around this [23:42:55] the techops folks think it's gross, but it generally works for me [23:43:42] I like my saltmaster because it doesn't mess with Puppet. [23:43:46] I wrote the code that will keep a self-hosted puppetmaster's git clone rebased on the ops/puppet upstream [23:44:17] I just leave puppet to do it's voodoo magic every so often and that works. [23:45:07] What does the base stuff in -ops/puppet do? [23:45:27] I know it does lots of stuff when it sets up an instance. [23:46:28] for a typical labs project instance it manages all of the config needed to talk to the ldap servers, nfs servers, etc that are provided as supporting infrastructure [23:47:05] that stuff changes pretty regularly which is why leaving puppet broken is generally a bad idea [23:47:17] So that's why my labs instances fail when any of that is down. [23:47:50] *nod* as soon as your ldap connection is broken your instance is bricked [23:48:04] you won't be able to login typically [23:48:52] there is some hiera magic you can do to try and install your ssh key for root that can help you get in to rescue things [23:49:38] see "passwords::root:extra_keys" here -- https://wikitech.wikimedia.org/wiki/Hiera:Google-api-proxy