[04:19:00] <wikibugs>	 10Traffic, 10Operations: HTML Dumps 429 error on RESTBase endpoints - https://phabricator.wikimedia.org/T255524 (10CDanis)
[09:23:18] <wikibugs>	 10Traffic, 10Operations: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368 (10ema) I found out more. This morning the list of affected hosts is shorter than yesterday:  ` cp2027.codfw.wmnet,cp5008.eqsin.wmnet,cp3050.esams.wmnet ` My theory is t...
[11:09:20] <wikibugs>	 10Traffic, 10Operations, 10affects-Kiwix-and-openZIM: HTML Dumps 429 error on RESTBase endpoints - https://phabricator.wikimedia.org/T255524 (10Kelson)
[11:10:33] <ema>	 mmh reading T255368#6227452 out loud I now think I might disagree with myself
[11:10:34] <stashbot>	 T255368: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368
[11:11:29] <ema>	 TS_LUA_CACHE_LOOKUP_HIT_STALE means that there is a stale object in cache for the given request, that ATS performed a conditional request against the origin for the object, and the origin replied with 304 "not modified" 
[11:11:54] <ema>	 so ATS concludes that the stale object is still fresh and returns it
[11:12:26] <ema>	 for TS_LUA_CACHE_LOOKUP_HIT_STALE requests we generate X-Cache-Status "miss", and I'm not entirely sure this is either right or wrong
[11:13:15] <ema>	 on one hand this is a hit (there's a corresponding cached object for the request, and we return it as-is to the client)
[11:13:40] <ema>	 on the other we did perform a request against the origin
[11:17:11] <volans>	 304-hit
[11:19:18] <ema>	 kinda-hit
[11:22:18] <ema>	 so varnish calls it a "hit", and after all from the point of view of the cache lookup that's indeed the case
[11:22:42] <ema>	 so I'm gonna go ahead and call those hit in ATS too
[11:26:38] <volans>	 :)
[11:46:04] <ema>	 cdanis: re: https://gerrit.wikimedia.org/r/#/c/operations/dns/+/605880/ -- can you add the task # please?
[11:46:18] <ema>	 cdanis: well first of all good morning and all
[11:47:42] <cdanis>	 hello hello :)
[11:47:50] <cdanis>	 I'm merging Arzhel's instead, he already had one with the bug #
[11:48:39] * ema nods
[14:18:46] <wikibugs>	 10Traffic, 10Operations: noc.wikimedia.org consistently 503s in eqsin and sometimes 503s in esams - https://phabricator.wikimedia.org/T255368 (10ema) >>! In T255368#6227452, @ema wrote: > we should try to reproduce `TE:chunked` being added to a stale object on 304 responses from the origin  I've written a simp...
[15:06:39] <wikibugs>	 10netops, 10Operations: No LACP info for cr2-esams:ae2 - https://phabricator.wikimedia.org/T253970 (10ayounsi) > This behavior is expected as the bundle is not running active with LACP enable. Instead is just a hardcoded aggregated bundle, in other words ; it is just Up with no participation of LACP for any so...
[15:45:03] <wikibugs>	 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul)
[15:45:12] <wikibugs>	 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul) p:05Triage→03Medium
[15:46:14] <wikibugs>	 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul) ` MD5 : 47ce46bc72ce069024a1694836961ed0 ` ` papaul@papauls-MacBook-Pro Downloads % md5 junos-srxsme-20.1R1.11.tgz MD5 (junos-srxsme-20.1R1.11.tgz) = 47ce46bc72ce069024a1694836961ed0
[16:47:48] <wikibugs>	 10Traffic, 10Operations: Configure varnish to use "Unconfigured domain" page for 404 Not Served (instead of generic error) - https://phabricator.wikimedia.org/T112316 (10Krinkle) 05Open→03Declined They are both based on the same template now. It is just the message heading that is suboptimal. But, I don't...
[16:57:50] <wikibugs>	 10HTTPS, 10Traffic, 10Operations, 10Toolforge, and 3 others: Migrate tools.wmflabs.org to https only (and set HSTS) - https://phabricator.wikimedia.org/T102367 (10bd808) a:03bd808 Last step is to set HSTS to 1 year and then close this.
[18:09:13] <wikibugs>	 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul) Version before  ` root> show version Model: srx300 Junos: 15.1X49-D170.4 JUNOS Software Release [15.1X49-D170.4] ` After upgrade  ` root> show version Model: srx300 Junos: 18.4R3-S2...
[18:09:39] <wikibugs>	 10netops, 10Operations, 10ops-codfw: codfw: rack/setup new srx300 (mr1) - https://phabricator.wikimedia.org/T255577 (10Papaul)
[18:23:01] <wikibugs>	 10Traffic, 10Analytics-Radar, 10Operations: Spammy events coming our way for sites such us https://ru.wikipedia.kim - https://phabricator.wikimedia.org/T190843 (10Astonmalie) I thought ru stand for Russia, this can just be a Russia version of wikipedia  This is my 2 cent though, i only assist students with [...
[22:48:57] <wikibugs>	 10Traffic, 10Operations: HTML Dumps 429 error on RESTBase endpoints - https://phabricator.wikimedia.org/T255524 (10CDanis) @Kelson I don't believe this affects Kiwix, this is about a different kind of dumps.  @RBrounley_WMF If you made your requests from a [[ https://wikitech.wikimedia.org/wiki/Portal:Cloud_VP...