[08:02:13] hello people! [08:02:37] cp3010 was frozen for some reason, so I depooled it and powercycled [08:02:50] will wait for you guys before taking any more action [08:37:28] elukey: frozen? [08:37:33] elukey: you were unable to login? [08:37:57] vgutierrez: unreachable from ssh + mgmt console [08:38:31] didn't check exactly what happened in the lost, just depooled + rebooted basically [08:38:42] *logs [08:39:18] ack, thx [09:06:17] I don't see nothing weird on the machine logs, last event appears to be at 06:01:01, Mar 22 06:01:01 cp3010 CRON[20696]: (root) CMD (/usr/bin/logster -o statsd --statsd-host=statsd.eqiad.wmnet:8125 --metric-prefix=varnishkafka.cp3010.webrequest.misc JsonLogster /var/cache/varnishkafka/w [09:06:22] ebrequest.stats.json > /dev/null 2>&1) [09:07:44] brb [09:29:26] good morning [09:30:05] hey nothing on fire [09:30:42] that means I can have breakfast today \o/ [09:32:30] ema: hahaha right [09:32:55] cp3010 looks weird though, nothing on ipmi-sel nor the logs [09:33:00] what am I missing? [09:51:00] let's see [09:53:10] so the server was rebooted ~ 2 hours ago [09:53:27] yup [09:53:31] Luca restarted it [09:53:33] elukey: anything interesting in mgmt console? [09:53:38] powercycled by elukey [09:54:01] apparently he wasn't able to reach the instance over ssh or mgmt console [09:54:07] together with 3007 and 3008, 3010 is running 5.1.3-1wm7 since yesterday [09:54:37] ema: empty blank screen, unresponsive to any command [09:54:41] I checked SEL, there were no failures events logged, just some stuff from 2012/2013 [09:54:53] yep that too --^ [09:55:01] (I tried racadm getsel) [09:55:03] which includes the fix for https://github.com/varnishcache/varnish-cache/issues/1799 and the OH ref leak fix [09:55:05] yup, same I've seen [09:55:31] I don't think that a wild varnish is able to crash the mgmt interface though [09:55:38] nothing interesting in kern.log [09:55:46] Mar 21 06:25:06 cp3010 kernel: [1024227.296698] Process accounting resumed [09:55:48] the server is six years old [09:55:51] Mar 22 08:00:56 cp3010 kernel: [ 0.000000] Linux version 4.9.0-0.bpo.6-amd64 (debian-kernel@lists.debian.org) (gcc version 4.9.2 (Debian 4.9.2-10+deb8u1) ) #1 SMP Debian 4.9.82-1~wmf1 (2018-02-19) [09:58:07] so was the mgmt interface also down? It is up now [09:59:38] vgutierrez: unreachable from ssh + mgmt console [09:59:48] apparently [10:00:22] no sorry that was me not explaining it well [10:00:46] ssh was unreachable, mgmt was but no sign of a working tty [10:01:00] ok [10:09:42] I'm gonna repool the host, it looks all good to me now [10:29:58] ema: btw, if you've a moment I need your wisdom O:) [10:31:15] comparing varnishreqstats VS varnishreqstats.mtail, no rush though [10:31:55] also if you're ok with it, I'm merging https://gerrit.wikimedia.org/r/#/c/415260/ (PyBal BGP icinga check) cause it's been hanging there a couple of weeks and there is no reason to delay it more [10:35:35] ship it! [10:35:40] <3 [10:46:29] we should keep a running log/ticket somewhere of traffic we see that looks "questionable" for some reason or other (as in, questions for application developers about why it looks so strange) [10:47:30] my new addition for this morning from trawling problematic times in slowlog: [10:47:35] https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Gray's_Inn_Lane_Hand_Axe/daily/20151001/20151031 [10:47:59] it's fast now and generally non-problematic I think, but what's interesting is the response includes this header: [10:48:11] < content-location: http://aqs.svc.eqiad.wmnet:7232/analytics.wikimedia.org/v1/pageviews/per-article/en.wikipedia/all-access/user/Gray%27s_Inn_Lane_Hand_Axe/daily/20151001/20151031 [10:48:24] why are we sending internal-only URLs in header responses for public APIs? [10:49:43] also, I've seen more small-gzip-response hints today. I may look more at disabling it from the varnish side (i.e. blocking AE:gzip from varnish->app) [10:50:37] another interesting hint that I've gotten nowhere with yet: [10:51:06] some slow responses to regular wiki page accesses, have the following sort of pattern in their timings+headers: [10:51:34] time-apache-delay 0.101334 [10:51:51] time-fetch 61.724494 [10:52:07] response-Age 61 [10:52:38] I've seen this more than a few times, where the inexplicable gap from apache-delay->fetch timing is (roughly to the second) the same as the Age value in the response... [10:53:41] I'm not sure what to make of that. it may just be "normal" if the headers at least arrived quickly (varnish records age as zero on quick header reception, spends 61s fetching, then serves the object with Age=61?) [10:54:58] interesting [10:55:33] we can double-check if that is indeed varnish's behavior with vtc [10:57:15] although if that's the case it sounds like a bug (age being the time an object has been in cache, if it hasn't been fetched yet how can it be in cache?) [10:57:23] well [10:58:02] I wouldn't say it's a bug, more a question of interpretation, and I think varnish's interpretation there is reasonable. [10:59:21] it's looking at it as: "The applayer sent no Age header or Age:0, we record Age:0 immediately and assume a fresh object has been created. That it took 61s to finish moving bytes across the wire afterwards before we could then respond to the client is irrelevant. 61s has now passed since the object's birth and thus the age is now 61" [11:00:10] of course all of the above only makes sense if we're store-and-forward, but we're defaulting to streaming... [11:00:32] if we're streaming, it would stream through the Age:0, and spend 61s slowly streaming things through, right? [11:00:58] so it seems like this can only happen when it's not a wire-speed-limit on slowly transferring content. [11:02:09] ema: a question I don't have the answer to in my head: in normal default streaming mode, does varnish forward response headers to the client immediately if the body stalls? [11:02:47] I know we get separate vcl callbacks that give us the chance to make decisions after response header reception and before fetching the body... [11:03:39] so surely, it doesn't forward the headers until after the callback where we get to edit them and make decisions. I wonder if it immediately sends output headers afterwards though, or waits until at least some body bytes are ready to stream through... [11:08:17] sigh.. I assumed the behaviour of check_prometheus (bash) and check_prometheus.py was the same.. and of course I was wrong [11:15:04] *check_prometheus_metric to be accurate [11:17:54] bblack: age is 0 regardless of the time spent to fetch the body https://phabricator.wikimedia.org/P6881 [11:24:56] ema: so then the reasonable interpretation of these slowlogs is that the applayer was setting that Age value? [11:25:21] if so it's still very odd.... [11:25:53] I'm assuming (perhaps incorrectly) that the applayer does headers then content separately [11:26:07] it would seem impossible that it would predict its own body-send timing [11:26:36] however! [11:26:40] but it also seems unlikely that if the applayer spent 61s spinning internally before it ever generated the header output, that it wouldn't be recording a low apache D= timing [11:26:47] with beresp.do_stream=false in vcl_backend_response, Age=2 [11:26:55] hmmm [11:29:16] bblack: re: content-location: http://aqs.svc.eqiad.wmnet:7232 - will ask to my team, could be some setting that needs to be tuned on the AQS side [11:29:53] elukey: well, I'd suspect this to be more of an RB-level thing and AQS just being an example [11:30:14] elukey: and they may well say "we need to return this as some unique key and that's the right value and clients are never expected to deref it", I donno [11:30:38] bblack: yep yep, but I'll open a task to investigate, it is weird [11:32:10] full vtc here for future reference https://phabricator.wikimedia.org/P6882 [11:37:20] ema: https://gerrit.wikimedia.org/r/#/c/421267/ ? [11:37:45] we've lived with the junky bad-gzip -> 503 for 500s thing for a long time anyways [11:38:25] but I've seen a fair number of these CL:0 (or CL<100) responses that are application-gzipped in the slowlog investigations targeting problem times, too, so it's kind of a shot-in-the-dark at that. [11:42:07] that would mean storing objects uncompressed on varnish-be at the backend-most layer, right? [11:44:57] > If the server responds with gzip'ed content it will be stored in memory in its compressed form [11:45:11] it shouldn't mean that, no [11:45:33] because we also turn on do_gzip elsewhere for a wide range of compressible mime types [11:45:43] (so varnish will compress it on the way in) [11:45:50] ah, right [11:46:04] the biggest risk is that there's a large jump in CPU% or something, from offloading applayer gzips to varnish [11:46:31] but I don't think gzip costs enough to matter, on the be-miss side of things [11:48:38] also https://gerrit.wikimedia.org/r/#/c/295372/ [11:48:51] ok yes, we do set do_gzip in wm_common_backend_response [11:48:51] for the cases where varnish is already the one gzipping, we tuned it there [11:49:28] if cpu% looks problematic, we could back down the settings or re-think (or finallly go package and deploy the cloudflare zlib improvements, if they haven't already trickled through upstream) [11:51:58] ship it! [11:53:41] yeah, if you're not doing related things, I might puppet-disable the fleet and see how it looks on a single eqiad text node first (since it only really has cpu impact there) [11:53:57] sure, go ahead [11:55:09] I was on upload@esams reboots for kernel upgrades but don't mind a coffee break :) [11:56:31] btw, I haven't been looking hard enough but probably should more... any differential for the numa_networking seting in recent troubles? [11:56:59] in the esams text case it's 3030+3031 that have it, the other 6 don't [11:57:12] (there's 2/N from each site+cluster altogether) [11:57:57] the non-numa networking text@esams hosts seemed to be equally messed up [11:58:49] (at the general OMG it's broken level, I haven't looked closely) [12:01:42] right, ok [12:03:14] ema: [12:03:19] https://grafana.wikimedia.org/dashboard/db/varnish-machine-stats?orgId=1&var-server=cp1065&var-datasource=eqiad%20prometheus%2Fops&from=now-1h&to=now [12:03:30] cp1065 only one, can see the +cpu% pattern there [12:03:39] also interesting is the matching +network recv [12:03:47] (since xfer is uncompressed from applayer) [12:04:44] anyways, seems sane enough, it's not going to suddenly melt all the things [12:06:15] https://gerrit.wikimedia.org/r/#/c/421273/ merging fix for prometheus_query.. the old (bash) check requires the query to be wrapped into scalar() [12:37:14] ema: this too https://gerrit.wikimedia.org/r/#/c/421275/ , but waiting till after puppetmaster stuff settles down [12:37:56] (I think it will kill a lot of useless slowlog noise so we can see overall patterns of true misbehavior clearly) [12:47:28] also interesting is the +hitrate increase seems to be real, across both upload+text, from the elimination of the healthy/grace stuff in vcl_hit. roughly from 95.3% global avg to 96.3% global avg on true-hitrate over a 24h period comparing same day-of-week a week before. [12:48:27] it sort of makes a rough sense, as we've elimiante a source of "return (miss)", but I wouldn't have expected an extra ~15m of grace-able hits to have such an impact (of course it's probably also the elimination of the related bugs, too) [12:49:03] https://grafana.wikimedia.org/dashboard/db/varnish-caching?orgId=1&from=now-7d&to=now&refresh=15m [12:49:19] ^ shows the overall pattern bump circa 2018-03-21 04:00 [12:50:21] (it's not as visible in the "disposition" graph, but I've confirmed it's real there too. actual hit/all goes up, miss/all goes down matchingly, and pass only drops slightly) [12:52:05] that's something like a net 20% drop in applayer requests between all clusters [12:53:33] wow [12:55:44] interesting [12:59:15] err, the 20% number was true-hitrate-centric and thus inaccurate [12:59:28] when including all the pass-traffic and looking at the true overall rates [12:59:37] it's more like a 15% drop in applayer-facing requests out of varnish [13:01:28] ah not even that, if I really get apples:apples same day-of-week like before [13:01:50] week-earlier: https://grafana.wikimedia.org/dashboard/db/varnish-caching?orgId=1&from=1521605089730&to=1521691507140&var-cluster=All&var-site=All [13:02:11] hmmm nope, copypasta [13:02:24] recent day: https://grafana.wikimedia.org/dashboard/db/varnish-caching?orgId=1&from=1521605089730&to=1521691507140&var-cluster=All&var-site=All [13:02:33] week earlier: https://grafana.wikimedia.org/dashboard/db/varnish-caching?orgId=1&from=1521000289000&to=1521086707000&var-cluster=All&var-site=All [13:02:55] so terminal-layer average to applayer is 9.7% -> 9.0% [13:03:10] more like a 7% drop for that comparison [13:04:22] (avg across all clusters' requests) [13:04:29] and it's very upload-biased [13:05:07] upload lost ~30% there, and text only ~3% heh [13:11:04] sigh.. we need to come with a way to benchmark graphite VS prometheus as metric backends [13:11:28] we rely on dashboards a lot [13:12:05] benchmark the query performance behind grafana you mean? [13:12:15] in general prometheus is going to be slower, it's a whole different data model [13:12:19] not the performance [13:12:28] the quality / accuracy [13:14:42] heh [13:14:46] so we need a way to validate the metrics sources, aka varnishstats VS .mtail programs [13:15:05] well I suspect a lot of it's going to be like the case I was blabbering about with ciphersuites [13:15:09] yup [13:15:27] but for varnish-caching, a pretty simple dashboard in terms of queries [13:15:34] prometheus-varnish-caching looks weird [13:15:42] and the data doesn't match [13:15:43] hammering on graphite randomly to get the graph shapes you want leads to statistically-invalid pretty things, and prometheus is a little more natural to get more accurate results [13:17:09] yeah... but varnish-caching is not hammered at all [13:17:16] it's just some rates and some counters over time [13:18:21] well [13:18:55] I have in the past even in simple cases tried to ask questions and/or look myself, and gotten confusing answers as to what .rate really means in various contexts around data sources from statsd or others, etc [13:19:28] I suspect there's some fundmanetal problems in our old pipelines for getting meaningful numbers without committing statistical sins [13:19:54] so we should embrace the new dashboards and not look back? [13:20:04] I have never been able to convince myself that for these existing graphite-based ones, I can actually numerically trace how it works all the way back to what true-live-samples should look like at any given second [13:20:36] there's various kinds of sampling/smearing/averaging going on at multiple layers on different time divisions, etc [13:25:54] so the best option should be comparing the output of the old (graphite) metric source VS the prometheus one [13:26:15] to ignore the statitics crimes commited across our monitoring infra [13:26:38] yeah maybe [13:30:25] https://phabricator.wikimedia.org/T184942 looked way easier than this *sigh* :) [13:33:23] lunch time, brb [13:41:30] yeah I've successfully managed to hide the open can of worms behind my back [13:44:34] and really, even if you think you've established a clear and statistically-valid pathway from "live OS stat" -> [...] -> "grafana graph" [13:45:08] then you have to consider what the live OS stat you're measuring really meant statistically, which is sometimes complicated and will thus muck up the rest of the pipline's validity as well [13:46:01] e.g. http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html [13:48:34] key quote: [13:48:38] "These three numbers are the 1, 5, and 15 minute load averages. Except they aren't really averages, and they aren't 1, 5, and 15 minutes. As can be seen in the source above, 1, 5, and 15 minutes are constants used in an equation, which calculate exponentially-damped moving sums of a five second average. The resulting 1, 5, and 15 minute load averages reflect load well beyond 1, 5, and 15 minutes. [13:48:44] " [13:48:59] #define EXP_1 1884 /* 1/exp(5sec/1min) as fixed-point */ [13:49:02] #define EXP_5 2014 /* 1/exp(5sec/5min) */ [13:49:05] #define EXP_15 2037 /* 1/exp(5sec/15min) */ [13:49:43] so, good luck stuffing that into some averaging/smoothing stuff in grafana and building something that's accurately statistically meaningful in a way that relates to other system graphs shown beside it [13:49:44] bblack: any ongoing experiments or can I carry on with reboots? [13:50:01] I don't know if the puppet agents are all re-enabled yet [13:50:14] (not from me, from the puppet ca work) [13:50:47] right, I see that cp3044 has puppet enabled (though it ran last time 99 minutes ago) [13:50:58] bblack: sigh^2 :) [13:53:32] my purpose in life is to crush other people's souls by shining light on the inscrutable madness that they thought was a well-ordered world they could understand :) [13:54:12] too bad that doesn't fit into a tshirt [13:55:40] puppet reenabled everywhere btw [13:55:47] yay! [13:55:48] <3 [13:59:30] yey.. after running puppet-agent on icinga the bgp session check is working as expected [13:59:33] :D [14:01:36] \o/ [14:05:21] I should at least make a task about the 200-OK-failed thing before I forget that one [14:18:11] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4072792 (10BBlack) [14:28:34] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4072782 (10Anomie) >>! In T190410, @BBlack wrote: > 1. Relatively-minor issue: It times out internally: there's a ~60s pause before any output is... [14:50:55] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4072879 (10BBlack) >>! In T190410#4072815, @Anomie wrote: > Define "times out internally". I mean this from the naive point of view of: when I... [15:20:43] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4072934 (10Anomie) >>! In T190410#4072879, @BBlack wrote: >>>! In T190410#4072815, @Anomie wrote: >> Define "times out internally". > > I mean... [15:34:58] 10netops, 10Operations: eqiad 10G ports needs - https://phabricator.wikimedia.org/T190364#4073014 (10faidon) I have only hunches and no data to back any of this, but I think ElasticSearch, Hadoop, WMCS, Backups, plus probably Ganeti and Kafka would be good candidates to go 10G-only. Kubernetes I could see it g... [15:37:58] 10netops, 10Operations: eqiad 10G ports needs - https://phabricator.wikimedia.org/T190364#4071230 (10Ottomata) I'd love to see stream processing from Kafka running in Kubernetes one day (pipe dream!), and that could be highish traffic. [16:19:07] staring at graphs again [16:19:40] this is the 20th, number of backend objects: [16:19:41] https://grafana.wikimedia.org/dashboard/db/varnish-failed-fetches?orgId=1&from=1521477686128&to=1521553407897&panelId=8&fullscreen [16:20:48] and the 21st: [16:20:50] https://grafana.wikimedia.org/dashboard/db/varnish-failed-fetches?orgId=1&from=1521542663000&to=1521676729327&panelId=8&fullscreen [16:21:51] now as bblack was saying earlier, we've kept on focusing on the varnish-be side because it's the one doing all the crazy things, but perhaps they are induced by some abnormal varnish-fe behavior [16:24:05] s/panelId=8/panelId=3/ in those URLs above to get to failed fetches [16:28:22] 10netops, 10Operations, 10cloud-services-team: modify labs-hosts1-vlans for http load of installer kernel - https://phabricator.wikimedia.org/T190424#4073206 (10RobH) p:05Triage>03Normal [16:44:52] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4073263 (10BBlack) >>! In T190410#4072934, @Anomie wrote: >>>! In T190410#4072879, @BBlack wrote: >> However, I find it a bit specious to use the... [16:53:07] <_joe_> uhhh I must resist the urge to answer on that ticket? [16:54:25] 10Traffic, 10Fundraising-Backlog, 10Operations, 10fundraising-tech-ops: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561#4073341 (10BBlack) Yeah I do have concerns here. It's going to take some time before I can loop back and explain them, but I just wanted to put the not... [17:17:24] 10netops, 10Operations, 10ops-eqiad: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960#4073423 (10Cmjohnson) Row A connections are complete cr1 xe-3/0/0 -> xe-2/0/44 #4776 cr1 xe-3/1/0 -> xe-2/0/45 #3452 cr1 xe-4/0/0 -> xe-7/0/44 #1985 cr1 xe-4/1/0 -> xe-7/0/45... [17:43:53] 10Traffic, 10Fundraising-Backlog, 10Operations, 10fundraising-tech-ops: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561#4073507 (10CCogdill_WMF) [17:44:15] 10Traffic, 10Fundraising-Backlog, 10Operations, 10fundraising-tech-ops: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561#4012170 (10CCogdill_WMF) Updating task as I want to update the subdomain in the request. [17:50:49] bblack: long shot, but do we have data on latencies that are acceptable for users or not? or a median/average latency we should aim for in general? [17:51:15] I remember a paper or blogpost written by google or youtube on the topic, but can't find it anymore [18:10:18] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4073589 (10Anomie) On the other hand, with all those different proxies that makes it much more likely one of them is going to throw its own error... [18:21:10] bblack: thanks to the services team the content-location issue of this morning should be fixed now! [18:27:42] XioNoX: eh there's a lot of wishy-washy data on that, but it's highly context dependent and somewhat subjective [18:28:07] XioNoX: either way, we're aiming for the best we can reasonably get given constraints! :) [18:28:35] bblack: yeah, it's for generating a pretty map, what latency should be green, what should be red, etc. :) [18:32:21] 10netops, 10Operations, 10ops-eqiad: Rack/cable/configure asw2-c-eqiad switch stack - https://phabricator.wikimedia.org/T187962#4073679 (10Cmjohnson) @ayounsi xe-2/0/44 -> cr1-eqiad:xe-3/0/2 #1984 xe-2/0/45 -> cr1-eqiad:xe-3/1/2 #3452 xe-7/0/44 -> cr1-eqiad:xe-4/0/2 2627 xe-7/0/45 -> cr1-eqiad:xe-4/1/2 346... [18:45:04] 10netops, 10Operations: Security audit for tftp on Carbon - https://phabricator.wikimedia.org/T122210#4073756 (10Dzahn) [18:48:29] 10netops, 10Operations: Security audit for tftp on install1001 - https://phabricator.wikimedia.org/T122210#4073763 (10ayounsi) [18:50:21] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4073785 (10BBlack) >>! In T190410#4073589, @Anomie wrote: > On the other hand, with all those different proxies that makes it much more likely on... [18:51:36] 10netops, 10Operations, 10ops-eqiad: Rack/cable/configure asw2-c-eqiad switch stack - https://phabricator.wikimedia.org/T187962#4073787 (10Cmjohnson) change cable number for cr1 xe- 4/0/1 to following xe-7/0/44 -> cr1-eqiad:xe-4/0/2 #3509 [19:04:49] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Define turn-up process and scope for eqsin service to regional countries - https://phabricator.wikimedia.org/T189252#4073815 (10BBlack) We've talked about this a bit this week. Basic initial steps of the plan at this point are: 1) T... [19:07:51] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4073831 (10Anomie) >>! In T190410#4073785, @BBlack wrote: > Yes, it's entirely possible any of the proxies might cause or record errors, and we'd... [19:08:03] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4073833 (10Anomie) [19:08:26] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4072782 (10Anomie) It's now clear to me that this task is a duplicate of T40716, so I'm going to close it as such. [19:16:38] 10netops, 10Operations, 10ops-eqiad: Rack/cable/configure asw2-a-eqiad switch stack - https://phabricator.wikimedia.org/T187960#4073864 (10ayounsi) [19:17:16] 10netops, 10Operations, 10ops-eqiad: Rack/cable/configure asw2-c-eqiad switch stack - https://phabricator.wikimedia.org/T187962#4073866 (10ayounsi) [19:49:51] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4073920 (10BBlack) >>! In T190410#4073831, @Anomie wrote: > Yes, there's a difference between getting "500 Something broke" and getting a data st... [20:58:10] 10Traffic, 10Operations: How is Varnish errorpage enabled for empty 404 text/html from mw/index.php?action=raw - https://phabricator.wikimedia.org/T190450#4074178 (10Krinkle) [21:01:42] 10Traffic, 10Operations: How is Varnish errorpage enabled for empty 404 text/html from mw/index.php?action=raw - https://phabricator.wikimedia.org/T190450#4074189 (10Ragesoss) @Krinkle thanks much! From that description, I'm guessing this won't affect many people. I was querying for arbitrary .json pages and n... [21:07:49] 10Traffic, 10Operations: How is Varnish errorpage enabled for empty 404 text/html from mw/index.php?action=raw - https://phabricator.wikimedia.org/T190450#4074212 (10Krinkle) a:05Krinkle>03None [21:51:17] 10Traffic, 10MediaWiki-API, 10Operations: Query API for rev props times out with an error message, but status is 200 OK - https://phabricator.wikimedia.org/T190410#4074369 (10Anomie) >>! In T190410#4073920, @BBlack wrote: > What exactly is the client going to do differently with `