[07:57:28] 10Traffic, 10DBA, 10MediaWiki-API, 06Operations: Someone is parsing all enwiki pages using the action api at a rate of ~2M pages/hour - https://phabricator.wikimedia.org/T162129#3174291 (10fgiunchedi) p:05Triage>03High I'm triaging as high since there's potential for an outage. Did the block or rate li... [07:58:34] 10Traffic, 06Discovery, 06Maps, 06Operations, 03Interactive-Sprint: Make maps active / active - https://phabricator.wikimedia.org/T162362#3174294 (10fgiunchedi) p:05Triage>03Normal [08:05:30] 10Traffic, 06Discovery, 06Maps, 06Operations, 03Interactive-Sprint: Make maps active / active - https://phabricator.wikimedia.org/T162362#3160389 (10Pnorman) Unless you take special measures two tile servers with the same style and data may render labels differently. Generally this is caused by queries w... [08:05:39] 10Traffic, 10DBA, 10MediaWiki-API, 06Operations: Someone is parsing all enwiki pages using the action api at a rate of ~2M pages/hour - https://phabricator.wikimedia.org/T162129#3174307 (10Marostegui) Looks like he stopped two days ago: https://grafana.wikimedia.org/dashboard/db/api-summary?orgId=1&from=14... [08:51:43] cp4005 has been running fine during the night with 4.9, I'm gonna go on with upgrading cache_upload [13:04:49] bblack, ema FYI I'm about to test the varnish switchdc task with a noop change in VCL if you don't have anything against or in the middle of something: [13:04:52] https://gerrit.wikimedia.org/r/#/c/347828/ [13:06:07] related executed code is: https://github.com/wikimedia/operations-switchdc/blob/master/switchdc/stages/t05_switch_traffic.py [13:06:09] I'm ok. I don't know if ema's in the midst of some 4.9 kernel update reboots, in which case 1/N caches might fail the agent run for being down [13:06:11] volans: go ahead, I've stopped the upload kernel upgrades meanwhile [13:06:19] and dry-run output is: https://wikitech.wikimedia.org/wiki/Switch_Datacenter/MediaWiki#output_6 (second part, from switchdc.stages.t05_switch_traffic) [13:06:20] well there we go :) [13:06:32] great, thanks [13:06:41] oh hmmm [13:06:57] it's not critical now, but you might want to alter the eqiad step to avoid cp1008 [13:07:04] (e.g. maybe an additional filter on *.eqiad.wmnet) [13:07:51] it's a test-server that uses production's role, but it's special and could very well be in an odd state that causes you to see false failures when you execute this, etc [13:08:05] volans: ^ [13:08:06] sure, let's avoid it [13:08:31] 'and not cp1008.wikimedia.org' I guess? [13:08:38] yes, but is it "fixed? [13:08:45] can I get it dynamically from something else? [13:08:58] I don't think so [13:08:59] role::authdns::testns ? [13:09:52] bblack: ^^ not sure if there is any "enforcement" that means it will always be that one [13:10:17] there's not any good role-based way to do it reliably [13:10:25] class parameters? [13:10:28] but all real cache hosts are in private networks, and it's public in .wikimedia.org [13:10:38] hence *.eqiad.wmnet as an easy filter [13:10:55] ok, or not *.wikimedia.org :D [13:11:22] then ema, let me fix this, feel free to continue with some kernel upgrade in the meanwhile [13:11:53] volans: OK please ping me when you're done [13:11:55] sure [13:12:24] also I'm forcing run puppet only in eqiad/codfw, so if you're doing esams/ulsfo there is no conflict [13:21:25] volans: I'm doing all DCs, but in any case I'd say it's better to avoid multiple concurrent work of this sort :) [13:21:40] yeah of course [13:26:33] volans: +1 on the *.wikimedia.org filter, only minor nit is that you might want to add a comment to remember why the filter is there in the first place [13:26:59] make sense, let me add that too, the commit message will be lost in history :) [13:28:03] done [13:37:32] volans are you about to start? [13:37:46] oh I guess you're in-progress [13:38:00] waiting for CI to submit, then I can proceed [13:38:08] but if you have other stuff you want to do first [13:38:10] I can wait [13:38:12] nope [13:39:38] rebasing the VCL change to now have to wait on CI during the test [13:39:56] seems like 1 blank line can skip CI [13:40:10] (and really, all our pre-staged stuff, can CI ahead of time and assume it's the same on rebase I think) [13:40:21] bblack: double check, is it ok to run puppet in parallel on all eqiad caches (8 servers)? [13:40:27] yes [13:42:24] ok, ready [13:42:32] volans: just a sec [13:42:52] sure [13:43:02] cp1072 is booting slowly because of T162612 [13:43:02] T162612: codfw hosts occasionally spend > 3 minutes starting networking.service with linux 4.9 - https://phabricator.wikimedia.org/T162612 [13:43:41] ema: sure let it also do a full puppet run [13:43:46] and let me know when ready [13:44:30] ema: I thought you guys were blacklisting uncore to fix that? [13:45:41] bblack: true, that was the plan [13:46:03] let's do that after the switchdc test [13:47:28] volans: done [13:47:59] ema: great, also the puppet run I guess [13:48:03] yup [13:48:21] proceeding then [13:50:06] running puppet now [13:51:29] so the diff is shown 4 times for each host [13:51:44] I guess we could go with run-puppet-agent -q, thoughts? [13:51:55] done [13:54:57] yeah, so, re: the 4x per host [13:55:34] cache::app_directors affects wikimedia-common template [13:56:02] the wikimedia-common template is templated-out twice on disk for the two different frontend/backend instances (with per-instance variation) [13:56:13] 10netops, 06Operations, 10ops-esams: cr2-esams FPC 0 is dead - https://phabricator.wikimedia.org/T162239#3175202 (10ayounsi) 05Open>03Resolved Juniper received the faulty part, > Thank you for returning your defective product in relation to your recently created RMA. This notification confirms that Juni... [13:56:28] and then we also have duplicates of everything templated out slightly-differently again for varnishtest [13:57:00] run-puppet-agent -q is fine I think, so long as we're absolutely sure the change applied when we get success back [13:57:37] we have the insurance that exit code was 0 [13:57:45] from run-puppet-agent [13:57:58] https://github.com/wikimedia/puppet/blob/production/modules/base/files/puppet/run-puppet-agent [13:58:03] last line [13:58:04] right [13:58:12] but that just means the agent succeeded, not that it applied the change we wanted :) [13:58:38] the race condition that scares me (and I saw it earlier this week in practice) [13:58:48] is that this sequence: [13:59:32] (on target nodes)# disable-puppet [13:59:42] (on gerrit)# C+2 -> Submit [13:59:51] (on puppetmaster)# puppet-merge # and wait for it to complete [13:59:58] (on target nodes)# enable and then run agent [14:00:22] does not gaurantee that the change you were working on actually gets applied to the target nodes [14:00:50] apparently even after puppet-merge completes, there can be a small window before the masters pick up all the on-disk changes and use them in new catalog compiles [14:01:05] so you can get a no-op success on the last step there. then wait a short while and try again and your change happens. [14:01:55] bblack: is there any way to know when the masters finished picking up the change? [14:02:10] I don't know what the boundaries are on that short while. I've only witnessed it and noticed it when the timing gap from puppet-merge completion to agent startup was <10s or so [14:02:33] but what's the upper bound there? if we're automating lots of things we'll eventually hit every corner case [14:02:45] ema: no idea! :) [14:02:48] ok, so then we could change run-puppet-agent [14:02:51] to use -t [14:02:58] yeah but even -t sucks right? [14:02:59] and check for exit code 2 [14:03:07] , an exit code of '2' means there were changes [14:03:12] from https://docs.puppet.com/puppet/3.8/man/agent.html [14:03:21] well technically it could be other changes too [14:03:33] so we could pass a paremter to run-puppet-agent that says expect changes [14:03:38] and it will fail if exit code is not 2 [14:03:39] and trying to trawl through all the outputs to "see what you expected to see" sucks too [14:03:42] ema: sure, that case too apply [14:03:55] the core of the problem is that puppet is so non-transactional about these updates [14:04:02] yeah ew know that [14:04:05] *we [14:04:32] it just gets to be even more of a problem when we automated around it tighter, vs a slower process where a human's at least trying to catch puppet's foibles [14:04:49] the other option is run-puppet-agent (without -q) | grep [14:05:01] not very reliable either [14:05:29] that points us even more to avoid puppet for those kind of changes that we want to automate :D [14:05:31] I wonder if there's a setting by which we know the time bound [14:05:41] e.g. some puppetmaster setting for "scan for disk changes every X seconds" or something [14:06:22] volans: can I resume the kernel upgrades? [14:06:24] we've also seen even worse related cases, where a change is half-picked-up [14:06:30] ema: sure! [14:06:34] cool, thanks [14:06:44] (e.g. the manifest change and related template change from the same commit, there's a time gap between puppet picking the two up) [14:06:56] ema: sorry I though was explicit the "done" before :) [14:07:20] volans: yeah, I just wanted to double-check :) [14:07:37] bblack: I'll try https://gerrit.wikimedia.org/r/#/c/347848/ by hand on the next couple of hosts [14:07:38] bblack: it would be nice to have puppet log the commit applied [14:07:54] I don't think puppetmaster is even aware that it's reading from git [14:08:08] it just thinks it's reading from a filesystem, right? [14:08:51] bblack: probably not (git aware), but we could have a post-hook that set's some variable to the git commit hash and have the clients log it [14:09:36] a post-what-hook? [14:09:42] (catalog compilation?) [14:09:44] puppet-merge [14:10:24] a puppet-merge hook doesn't tell us much about the puppetmaster process's state though [14:10:24] I need to check, but we do git-stuff there [14:10:38] no but if it set's a variable in hiera for example [14:11:06] or another local file that is included in the puppet compilation process [14:11:19] probably step 1 is even understanding what causes the delays (if it's a timer in puppetmaster to check changes, or maybe it uses inotify but we need to sync changes to disk better?) [14:11:21] and like in base have info() of that value [14:11:50] someone surely has studied this problem and filed complaints/bugs before [14:12:21] you think puppet devs ever received complaints? Impossible! [14:13:16] 10Traffic, 06Operations: Fix broken referer categorization for visits from Safari browsers - https://phabricator.wikimedia.org/T154702#2920977 (10Astinson) +1 to @Nuria 's comment: I think the main concern here from @DarTar and me is that external websites need to have some sort of awareness where the dark tr... [14:13:22] lol [14:17:43] puppet code deploy --wait production [14:18:01] ^ apparently this is from puppet enterprise stuff [14:20:24] is there even a place where pupptmaster would log such a thing [14:21:48] bblack: OK to merge https://gerrit.wikimedia.org/r/#/c/347848/? cp2005 and cp1063 booted fine with the module blacklisted [14:25:05] bblack: which thing? [14:25:18] volans: that it had noticed some changes on the FS or whatever [14:25:43] ema: maybe? I was looking at modules/base, I don't think it does any magic beyond pushing the modprobe.d file [14:25:55] doesn't it need to rebuild initramfs afterwards or something like that? [14:26:01] (or affect grub cmdline?) [14:26:23] I guess so long as this is loaded relatively-late, maybe not [14:26:28] bblack: I think it affects udev, adding a line should be enough [14:26:39] not sure, I can check it, but for now I'm ok to leave the puppet run verbose for the switchover [14:26:40] ok [14:26:47] x2 [14:27:18] the module hasn't been loaded on cp2005/cp1063, so it should have worked fine :) [14:40:04] so, re: the non-MW parts of https://wikitech.wikimedia.org/wiki/Switch_Datacenter [14:40:25] 1) I fixed up the Traffic section for the pre/post work there, it's pretty simple, two commits and their reverts [14:40:45] (that's at: https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Traffic ) [14:41:34] you used "run-puppet-agent -q" :-P [14:41:40] yup! [14:42:09] in this case, the changes aren't actually-critical [14:42:17] yeah 30m later at most it's ok [14:42:20] if a node fails to apply them, they'll apply in 30 minutes or less from cron and all's fine [14:43:32] 2) Swift - that section needs at the very least a much simpler rewrite, but I'm still not clear on whether it has any inter-dependencies outside of traffic or not, need to sync up with godog on what steps to execute there [14:44:09] 3) "Services" - I think this section as we delve into it is suffering from a services-services-services problem like labs-labs-labs [14:45:07] the overview outlined there now is just about the services in discovery::services, but also notes some of them are varnish-level services too, and then (I think unecessarily) links the varnish and dns-disc parts together as one sequence of steps [14:45:22] there are also "services" in varnish that are not in dns-disc, too [14:45:47] to simplify the picture [14:45:51] since dns-disc "services" are a/a anyways... [14:46:17] I think we can split that up into one whole stage of "move varnish-level services to codfw only as appropriate" and then a second stage of just the dns-disc changes for those services [14:46:26] I know that joe wanted to do a quick bash with all the services commands [14:46:47] I guess to change the confctls [14:46:53] for the discovery ones [14:47:04] right, but the ones with varnish-level changes are commits, which are blended into it in the current doc [14:47:09] I think they don't have to be [14:47:43] we can do a "Traffic Services" that does all the varnish-level commit-based switching of services that varnish uses as a seperate stage [14:47:56] and then "Services" is just the confctl for dns-discovery for the internal discovery-based services [14:48:17] (sequentially - finish up all Traffic Services then run Services) [14:49:12] yeah I guess too is better to decouple them [14:49:26] they're not for the same sets of services anyways [14:49:35] some are varnish-only, some are dnsdisc-only, some are both [14:49:59] bblack: I'm about to jump in an interview, sync up maybe after the switchover meeting? [14:50:15] godog: ok [15:04:35] bblack: re:run-puppet-agent -q, to be completely fair and make clear pro/cons, even without the -q if the race condition happens we're a bit screwed given that the script will continue running puppet on the other DC [15:06:24] we'll be aware of it looking at the output/scrolling it, but still we'll had already proceeded into the next command [15:08:54] there's no way to insert a pause with confirmation? [15:09:11] if needed sure, we have already one to ask to merge the commit [15:09:31] yeah I mean between the 2x DC puppet enable/run [15:09:54] yeah I mean ofc we can add another one :) [15:10:02] the whole reason for that split, is if puppet apply from the second DC happens before it's definitely-applied everywhere in the first DC, it will cause user-facing 508s [15:11:20] the other option could be what I said before, make a parameter to run-puppet-agent to ensure that changes were applied, even if it doesn't work in the general case, we know that this is the only commit of the switchover [15:11:34] and we could add a force puppet run just before the RO period on the same hosts [15:11:51] yeah [15:11:56] that's reasonable too [15:12:05] but then to be devil's advocate, the race condition could happen before [15:12:17] and at the second run the changes applied are the previous ones and not the intended one [15:12:32] "before" in the sense at this first run before the RO period [15:12:57] well hopefully we'll have a period of puppet quiescence before we step into this procedure's pre-steps anyways [15:14:12] another option, anything I could grep in the varnish config to ensure the intended config was applied? [15:15:21] I think that gets complicated [15:16:35] :) [15:17:05] there is no: varnish-tell-me-your-routing-path command? :D [15:17:36] :P [15:17:47] another random topic on the wikitech page: [15:17:55] Traffic: Pre-switchback in two phases: Mon May 1 and Tues May 2 (to avoid cold-cache issues Weds) [15:17:58] MediaWiki: Wednesday, May 3rd 2017 14:00 UTC (user visible, requires read-only mode) [15:18:01] Services, Elasticsearch, Swift, Deployment server: Thursday, May 4th 2017 (after the above is done) [15:18:08] ^ that's the switchback ordered list of things [15:18:45] other than deployment server, the other 3 at the end there (services, es, swift) are happening after MW switches back? [15:19:03] bblack: OT tell me when you've finished to edit the wikitech page, I have some changes to do too ;) [15:19:03] this seems to be the opposite of how we did things on switchover, in an "unwinding" sense [15:19:13] volans: done for now [15:20:15] oh no, I had that backwards by the time I finished thinking about it heh [15:20:27] it's just dpeloyment server that seems out of unwinding order [15:20:40] also traffic [15:20:44] should happen after MW [15:20:55] yeah but it's actually independent [15:21:22] I could make the traffic switch today and it makes no "functional" difference to users or what data flows into which apps in which DCs [15:21:34] yeah [15:21:47] ditto for deployment tooling I guess [15:21:59] although I don't know if that's true in the details [15:22:14] (e.g. deployment scripts that send maintenance API requests to the servers or something) [15:24:05] if you want, we can move the traffic switchback days to N+1 and N+2 (after MW) [15:24:21] it doesn't really matter when they happen, just that they're spaced out to avoid the cold cache issue [15:24:37] but I figure this way around lets us declare sooner that we're back to normal state [15:24:51] absolutely the same for me [15:25:41] eh leave it then, plenty of other things to discuss! :) [15:26:02] yeah! [15:31:29] upload ulsfo fully upgraded to 4.9, 19 hosts left in the other 3 DCs [15:57:59] volans: so in the MW/RO stuff the "merge the varnish patch This is not covered by the switchdc script [15:58:02] " [15:58:09] do you want me to prep that patch and insert a link there? [15:58:23] bblack: no [15:58:28] first, just updated the page [15:58:41] second the switchdc pause and ask the operator to merge+puppet-merge [15:58:53] the patch should be already done by jow [15:59:04] I'll review the links to the patches after the meeting [16:52:59] volans / bblack swift instructions LGTM, thanks! [16:53:14] bblack: re: the actual time of the day for traffic (and swift?) do you have any preference? [16:53:45] so no more salt in that page! \o/ [16:56:13] "we've successfully replaced salt with cumin" #diet [16:56:22] lol [16:58:08] OK, 27 upload hosts upgraded to 4.9, 12 left [16:58:53] slowly but surely less and less 4.4 https://grafana-admin.wikimedia.org/dashboard/db/kernel-deployment?orgId=1&from=now-7d&to=now [16:59:24] yay! see you all tomorrow [17:00:38] cya ema [17:09:43] 10Traffic, 10DNS, 06Operations, 06Services: icinga alerts on nodejs services when a recdns server is depooled - https://phabricator.wikimedia.org/T162818#3175699 (10BBlack) [17:25:00] bblack: for swift I was thinking 15 UTC, did you have a time in mind for traffic on the 18th? [17:26:54] I don't have any specific times in mind, no. whatever works for you works for me [17:27:45] godog: are there some other steps that need to happen either before the (temporary) active/active state, or before/after the switch to codfw-only? [17:28:01] godog, bblack: FYI I saw a lot of cronspam emails with: No such file or directory: '/tmp/vhtcpd.stats' [17:28:15] that's not good [17:28:54] oh, prometheus checks, probably on freshly-restarted cache machines [17:28:56] that kinda makes sense [17:29:34] the check with "systemctl is-active -q vhtcpd &&" isn't quite enough, basically [17:29:39] should check the file exists, too [17:29:46] bblack: not this year no, the mw-config patch we merged last year in the middle is now the default [17:30:07] from a quick check seems all 4.9 [17:30:09] (as vhtcp only outputs the file every 15s, and that means it's not there for the first 15s the daemon is up) [17:30:10] bblack: I've tentatively put swift at 15 and traffic at 16 (UTC) on the 18th [17:31:21] godog: if there's truly nothing else to coordinate but "do the traffic bit", we can also just consider swift to be another random cache backend service, which are all being handled together in: https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Traffic_-_Services [17:33:02] (the steps are the same for all of them and swift, just right now we have swift broken out as a separate set of commits and separate steps, instead of merged up into that unified set of commits/steps for all the others) [17:34:02] ok, I'm about to join SoS but I'll take a look later [17:34:12] ok [17:35:15] this is really not an important service, but i CAN switch planet1001 to planet2001 and it'd be active/active [17:35:27] i wonder if i should add that somewhere [17:36:22] mutante: on that whole topic... [17:36:37] basically, I think there are a number of different cache_misc services that are ready to do that [17:37:11] i wonder what will happen to phab and gerrit [17:37:11] I just don't have time to enumerate and validate them all (we can check mechanically if they seem to have a similar dns hostname or site.pp role in codfw, but even then that doesn't mean all issues are solved to allow it to sanely be active/active) [17:37:54] if there's some that we're *sure* are ready for active/active traffic right now today, we can set them a/a right now in cache_misc, and add them to the lists and commits for codfw-only for the switchver too [17:38:16] re: phab and gerrit, they'll just stay operating in eqiad like everything else we're not including in testing [17:38:43] ok, both phab and gerrit alreayd have 2001 counterparts but afaict they are just "almost ready" [17:39:30] right, if it's active/passive warm-standby stuff with some steps needed to handle the transition, basically we haven't planned for those to be in scope and it's too late to sort it out now, so they stay in eqiad this go-round [17:39:36] so when i listened to the talk about switchdc there was the part about the discovery zone in DNS [17:39:44] but that isnt there yet, is it? [17:39:49] it is [17:39:59] the discovery dns stuff is only for internal though, not public-facing [17:40:26] look at the bottom of the wmnet zonefile for the entries [17:40:54] oh! "disc-*" got it now [17:41:04] so i could add planet there i suppose [17:41:15] no, I don't think planet goes there [17:41:26] it's not an internal service that other internal services are consuming [17:41:50] the dns discovery stuff is all for internal service<->service traffic within .wmnet [17:42:02] aha, ok, so that is a change in misc_web varnish then? [17:42:09] right [17:42:17] in hieradata/role/common/cache/misc.yaml [17:42:17] will look at making that one [17:42:20] alright [17:42:47] ah, looks like i just add a second backend [17:42:55] you can see the example with e.g. "noc" or "pybal_config" in cache::app_directors [17:42:58] right [17:43:02] ok, cool [17:43:29] when you set that, it's active/active in a public-facing sense (as in EU users might hit the eqiad backend and Asia users the codfw backend and such) [17:44:19] ok, i just need to enable crons on both and that should be fine [17:44:22] (and some horribly-borked corner case users might flip-flop the two backends) [17:47:43] some directors have the "1001" in their name, but probably shouldnt if they have multiple backends [17:48:04] yeah :) [17:48:39] graphite is in an odd state too :) [17:48:58] but I'm basically just leaving it all alone for now, no time to sort out exactly what each backend is capable of in this sense and fix them all [17:49:45] yep, *nod* and you dont want many changes right before switch either [17:50:30] but sometime later in the quarter, after the switchback, we should probably audit all the cache_misc backends and set them up properly where it makes sense [17:53:52] yup [17:57:35] bblack: yeah I think swift can be effectively be considered under 'traffic - services' but I'd argue for keeping separate reviews and puppet runs, on the basis that cache upload has immediate user visible effects if sth goes wrong [18:00:51] godog: ok that's fine [18:01:18] that little special snowflake :-P [18:02:15] yeah the rest of them have user-visible impact too, but it's not much trouble to do things either way [18:04:54] *nod* [18:05:01] ok I'm off, see you tomorrow! [18:05:05] cya [18:40:06] hmmm ores [18:40:37] I see we're doing active/active for it for inter-service stuff. it also has a public endpoint through cache_misc at ores.wikimedia.org [18:40:51] do we want that a/a and/or switched for the switch-testing? [19:39:11] 10Traffic, 10MediaWiki-Cache, 06Operations, 06Performance-Team: Duplicate CdnCacheUpdate on subsequent edits - https://phabricator.wikimedia.org/T145643#3176241 (10aaron) 05Open>03declined The rebound purge is deliberate and hard to de-duplicate in any case (unless two purges came in at the same time a... [20:11:36] 10netops, 06DC-Ops, 06Operations, 10ops-codfw: setup wifi in codfw - https://phabricator.wikimedia.org/T86541#3176312 (10RobH) p:05High>03Normal a:05faidon>03ayounsi I chatted with @ayounsi about this via IRC. He is now aware of this pending task, though it isn't high priority. Basically codfw ha... [20:12:24] 10netops, 06DC-Ops, 06Operations, 10ops-codfw: setup wifi in codfw - https://phabricator.wikimedia.org/T86541#3176317 (10RobH) [20:13:02] 10netops, 06DC-Ops, 06Operations, 10ops-codfw: setup wifi in codfw - https://phabricator.wikimedia.org/T86541#970935 (10RobH) [20:13:24] 10netops, 06DC-Ops, 06Operations, 10ops-codfw: setup wifi in codfw - https://phabricator.wikimedia.org/T86541#970935 (10RobH) [21:13:21] 10Traffic, 10DNS, 06Operations, 06Services (watching): icinga alerts on nodejs services when a recdns server is depooled - https://phabricator.wikimedia.org/T162818#3176556 (10mobrovac) [21:44:01] should get rid of those IP alerts on channel: https://gerrit.wikimedia.org/r/#/c/347984/ [21:44:33] eh, no, i need to add a parameter, but yea [21:46:03] no i don't. it's already there:) it's just plural instead of singular. should just work [22:40:34] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Operations, and 3 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3176850 (10DStrine) [22:42:33] 07HTTPS, 10Traffic, 06Operations, 10Wikimedia-General-or-Unknown, 07JavaScript: Use Upgrade Insecure Requests on Wikimedia wikis - https://phabricator.wikimedia.org/T101002#3176879 (10Krinkle) >>! In T101002#2500137, @BBlack wrote: >>>! In T101002#1326438, @Krinkle wrote: >> This header currently results... [23:00:08] 10Traffic, 10DNS, 06Operations, 06Services (next): icinga alerts on nodejs services when a recdns server is depooled - https://phabricator.wikimedia.org/T162818#3176931 (10GWicke) [23:25:03] 10Traffic, 10DNS, 06Operations, 06Services (next): icinga alerts on nodejs services when a recdns server is depooled - https://phabricator.wikimedia.org/T162818#3177050 (10GWicke) There are big drops in *action* API backend requests and huge spikes in latency around both times: {F7514011} The same latenc... [23:30:56] 10Traffic, 10DNS, 06Operations, 06Services (next): icinga alerts on nodejs services when a recdns server is depooled - https://phabricator.wikimedia.org/T162818#3177056 (10BBlack) FWIW - I did the same depooling (for reinstalls) in codfw this afternoon, and there was no impact in that case. So this seems...