[00:49:10] 10Domains, 10Traffic, 10Operations, 10WMF-Legal, 10Patch-For-Review: Move wikimedia.ee under WM-EE - https://phabricator.wikimedia.org/T204056 (10Dzahn) a:05Dzahn>03tramm [08:51:04] ema: do you know if it's ok to merge DNS patches and if there is any particular procedure I should follow given we're in the middle of a gdnsd upgrade? [09:20:18] ah e.ma is not around, I'll wait for bblack confirmation ^^^ (it's not urgent, can wait the afternoon) [10:24:05] volans: should be fine, no particular procedure [10:25:03] although, if it's a config patch, I'd like to know so I can watch the details and make sure everything went ok [10:25:17] (but it should!) [10:37:49] bblack: thanks, it's https://gerrit.wikimedia.org/r/c/operations/dns/+/461919 [10:39:20] ah just zone data [10:39:27] should be fine! [10:39:31] yes pretty trivial :) [10:40:08] I mostly wasn't sure if the authdns update script should be launched from a specific host (old vs new version_ [10:41:16] if you prefer to merge+deploy it feel free [10:50:16] volans: no go ahead, it should be fine [10:50:24] ok, thx [10:50:38] there are currently 4 hosts instead of the usual 3, should be the main thing you even notice [10:52:11] yeah, multatuli too [10:52:15] all done [11:12:50] if I was given access to a time machine, I think eliminating systemd at its inception might be on my top 10 list of history edits :P [11:13:15] rotfl [11:13:32] still creating headaches to make all this work? [11:16:01] no new headaches lately, just still lamenting all the past headaches as I re-re-re-review all my related code for stupid bugs, and re-read systemd source to make sure I really know what its doing too. [11:16:50] (because why on earth would they document critical behavior in a way that's useful to actual developers? Just sorta gloss over it, so you have the freedom to make random future changes and blame the app) [11:16:53] eheh, how close will you need to keep an eye on systemd development from now on to make sure it will keep working? [11:17:04] who knows [11:17:37] would be nice to have some sort of automatic CI that for each new release of systemd runs the tests [11:17:43] it's clear the code I'm relying on is intentional and intended anyways, but the documentation of the code's behavior is lacking [11:18:37] also, technically the systemd end of things is racy, but probably not in a way that will ever crop up in realistic scenarios [11:18:42] the API design is racy for that matter :P [11:19:00] ofc, why not :) [11:19:50] the designed-in race here is that there's interactions inside of systemd's init proceess between: (a) being told over $NOTIFY_SOCKET of a main pid switch for a unit and (b) processing SIGCHILD for an existing mainpid exiting. [11:20:23] ok [11:20:34] the only thing they do in service of trying to resolve that race, is both are processed by the same eventloop system, and they set a higher priority on socket traffic than SIGCHLD-handling [11:20:54] 10Domains, 10Traffic, 10Operations, 10WMF-Legal, 10Patch-For-Review: Move wikimedia.ee under WM-EE - https://phabricator.wikimedia.org/T204056 (10tramm) Our preferred hosting provider doesn't have separate nameserver service for domains not hosted in their system. But as I see domain wikimedia.ee is mana... [11:21:49] the gdnsd handoff process, from systemd's perspective: new daemon sends MAINPID notification over $NOTIFY_SOCKET to systemd, then new daemon tells old daemon (over gdnsd controlsock) to shut itself down, and it has to do some orderly shutdown things like terminating threads and waiting out hald-fone TCP transactions, cleaning up resources... then exits. [11:22:26] but technically, the kernel provides no gaurantees about how fast a unix domain socket message will be delivered. It's theoretically possible the old daemon could exit before epoll bothers to tell systemd the notification is ready. [11:22:48] could old pid gdnsd check that systemd at this time has changed the main PID alredy to the new one? [11:23:18] not that I know of. possibly by using D-Bus? [11:23:55] systemctl status :-P [11:23:57] * volans hides [11:23:57] the simplest fix would've been to make it possible to do synchronous transactions over the $NOTIFY_SOCKET, so the new daemon could send "MAINPID=N" and *wait* for systemd to ack the message before terminating the old daemon [11:24:10] but it only has an async fire-and-forget interface [11:24:20] ehehe, not much you can do about it [11:25:41] anyways, the race is annoying, but odds are probably extremely high a SOCK_DGRAM single-packet AF_UNIX socket message goes through, before we can do all the other crap (talk to another daemon over yet another control socket, and it go through all of its shutdown sequence, then have the kernel send the SIGCHLD over to systemd. [11:26:23] and if the timing is even close, they'll happen in the same eventloop iteration and the socket will get priority and things will be fine. the awful, unlikely, but technically-possible outcode is SIGCHLD gets delivered a whole eventloop cycle before the message is noticed [11:26:59] I agree that the race is pretty unlikely to happen, was just wondering if there was a nicer way to get the equivalent of "systemctl show -p MainPID $service" and just check it was not anymore equal to current PID [11:27:02] s/outcode/outcome/ [11:27:39] there isn't a simple way, that I've found yet. [11:28:02] executing an external command is outside the complexity budget in this case I think, given all the trickiness already around forking/threading. [11:28:26] I think D-Bus is an option, but I also suspect it will be a highly-annoying one if I don't want to link some bloated library. [11:28:35] right [11:29:01] 10Domains, 10Traffic, 10DNS: redirect wikipedia.gr to el.wikipedia.org - https://phabricator.wikimedia.org/T205077 (10geraki) [11:29:30] maybe there's some better hack though, that can be inferred as a heuristic safety net [11:29:46] like looking at the cgroup ID of some pid involved in the process or whatever, something along those lines [11:30:26] anyways, stupid unlikely races aside, let's talk about documenting behaviors! [11:30:36] indeed! [11:31:30] as best as I can tell, MainPID-switching is intended. I think systemd even sometimes can infer MainPID switching of a unit. [11:32:15] and looking at the code, they go through well-commented/interfaced code that's clearly intended to support switching MainPIDs, by having another process send a MAINPID=N notification as described above. [11:32:48] so it looks something that should be stable [11:33:14] there's also a bunch of sanity-checks built in that could stop it: the notification has to be allowed from the new source (NotifyAccess=All), the alien mainpid thing, the new pid has to already be in the right cgroup (we are), etc... [11:33:45] looking at the code, the intended interface and rules for how to switch mainpids like this are very clear, and seem stable over some past revs I've looked at. [11:34:01] but if you just look at the documentation, there's really nothing on this subject at all. [11:34:55] sending MainPID over $NOTIFY_SOCKET is documented as a way to inform of initial mainpid (I guess in case your daemon forks in mysterious ways at startup that systemd can't follow easily) [11:35:26] but the whole "Here are the circumstances under which you can safely switch mainpids at runtime, and the rules you must follow" is completely undocumented territory in the docs. [11:35:37] wow [11:37:00] anyways, double-checking all doc refs now, but so far nothing useful [11:39:37] but yeah, maybe if they had bothered to document how to switch mainpids, they'd have realized while writing the docs that (a) it's unrealistic for many complex multi-threaded software and (b) it's racy :P [11:40:13] lol [11:40:16] yeah [11:42:46] even if we stick with much of their stupid design and previous decisions about cgroups, etc, a much smarter way to fix the two core issues would be: (1) there should be a way to send the new MainPID that's synchronous and ACKs it + (2) Some way to launch ancillary commands in the unit's real cgroup that can become mainpid, without the currently-running daemon having to fork them. [11:42:56] 10Domains, 10Traffic, 10DNS, 10Operations: redirect wikipedia.gr to el.wikipedia.org - https://phabricator.wikimedia.org/T205077 (10jijiki) [11:43:11] e.g. (2) could be a flag for ExecReload that lets it behave that way (launch an independent daemon, which starts out in the right cgroup context to take over) [11:43:17] or a different ExecFoo for it, whatever [11:43:49] avoid the necessity of an already-running daemon having to fork would expand the useability a lot [11:46:37] I guess, if I were trying to patch a much more complex daemon to grapple with these issues, I might take the approach of having it fork off its future replacement very very early while it's safe, and have that replacement just sit around waiting for a notification from the "real" daemon [11:47:00] or fork off a tiny manager process to sit in the cgroup, which can be notified to fork->execve() new copies of the daemon [11:47:14] yeah I was thinking the latter [11:47:21] seems more future-proof [11:47:45] yeah, I had structured gdnsd's stuff somewhat closer to that approach, during an earlier attempt at this I threw away [11:48:02] where the current "main thread" or whatever split itself into two processes, one a daemon manager and one the real runtime. [11:48:25] which is sort of like "Hey let's reinvent a tiny slice of systemd as an abstraction layer between broken-systemd and what we need" [11:48:31] but totally agree that all this overhead would be unnecessary [11:48:43] if only systemd was a little bit more manageable [11:48:51] eheheh [11:49:57] what galls me the most, is accomplishing these kinds of handoffs/takeovers worked/works fine on merely the assumption of standard *nix APIs working right and all the normal rules of how init systems work, and systemd came along and actively broke it. [12:09:12] XioNoX: whenever you get online, I'd like to move the esams authdns routing from eeden to multatuli [12:18:10] so esams authdns will be running the new gdnsd? [12:22:33] yeah [12:22:41] eqiad has been since earlier in the week, codfw isn't yet [12:23:21] eqiad and codfw were just software package updates, but esams authdns was still running debian jessie, so it needed an OS upgrade too [12:24:07] and given it takes the better part of an hour (maybe more!) to reinstall and repuppetize a node in esams, I figured it was better to use the multatuli spare and then switch, instead of re-routing esams dns traffic over to eqiad for a long period while reinstalling eeden. [12:29:11] yeah [12:29:18] does esams not have a local copy of the install media or something? [12:30:06] yeah, a bunch of things like that [12:30:49] the bulk of the data downloaded during the install process (packages, etc) all come from eqiad, and the puppetization all happens from eqiad (and puppet is really awful at network latency, it doesn't batch up anything) [12:31:13] hey, is this correct for IPs passed from varnish? https://github.com/wikimedia/ores/pull/267/files [12:31:17] oh right because the apt stuff is not locally mirrored, damn [12:31:37] i.e. are you setting the client IP using this header? HTTP_X_FORWARDED_FOR [12:32:08] that'll be an environment thing provided by uwsgi [12:32:28] which should come from the X-Forwarded-For header which varnish should pass in I think [12:33:26] Amir1: you probably want X-Client-IP if you're looking for a WMF-specific solution. Parsing X-Forwarded-For sanely is hard anywhere, and especially here. X-Client-IP is the "real" client IP after Varnish has processed XFF and related stuff. [12:33:56] make sure you don't allow connections in from untrusted hosts to that layer [12:34:48] Thank you a lot [12:34:54] let me amend the patch [12:37:41] Krenair: I think ores is only reachable on the internal network, or via-varnish for public access [12:37:51] ok [12:39:01] bblack: I amended the patch, can you double check if it seems right to you? https://github.com/wikimedia/ores/pull/267/files [12:39:53] lgtm fwiw [12:41:21] Thanks! [12:42:06] 10Domains, 10Traffic, 10DNS, 10Operations, 10User-jijiki: redirect wikipedia.gr to el.wikipedia.org - https://phabricator.wikimedia.org/T205077 (10jijiki) a:03jijiki [12:48:35] Amir1: yeah seems sane to me too, at least at the logical level, but I'm not great at reviewing python in my head yet :) [12:49:42] That's still pretty useful. Thank you. I need to deploy this right now. [12:51:03] bblack, so my gdnsd install has upgraded and fails to start :/ [12:51:26] Sep 21 12:50:16 deployment-certcentral-testdns gdnsd[30536]: DNS listener threads (1 UDP + 1 TCP) configured for 0.0.0.0:53 [12:51:26] Sep 21 12:50:16 deployment-certcentral-testdns gdnsd[30536]: DNS listener threads (1 UDP + 1 TCP) configured for [::]:53 [12:51:26] Sep 21 12:50:16 deployment-certcentral-testdns systemd[1]: gdnsd.service: Main process exited, code=exited, status=42/n/a [12:51:43] yeah the systemd journal view tends to be buggy [12:51:46] what does syslog say? [12:52:11] (also, upgraded from what rev to what rev?) [12:52:59] status=42 always has some syslog output that goes with it about the error, but systemd is notorious for ignoring the final important log message output of a dying process :P [12:53:01] ah yeah there's more there now [12:53:09] s/ now// [12:53:46] which problem is it? there's a couple of known cases recently-patched [12:53:46] It would previously have been using a custom build I made and it's now on 2.99.42-beta-1+wmf1 [12:54:10] cannot create rundir? [12:54:13] Sep 21 12:54:03 deployment-certcentral-testdns gdnsd[30656]: cannot open control sock lock at /var/run/gdnsd/control.lock: Permission denied [12:54:25] or that heh [12:54:28] root owns it [12:54:37] same with control.sock [12:54:41] so that's because you ran it as root before, you should "rm -rf /var/run/gdnsd" [12:54:53] but then it will fail in a new way probably [12:55:14] Sep 21 12:54:57 deployment-certcentral-testdns gdnsd[30699]: mkdir of run directory '/var/run/gdnsd' failed: Permission denied [12:55:36] should I recreate that and make it owned by gdnsd? [12:55:38] right, that's fixed in https://github.com/gdnsd/gdnsd/commit/fdc9f039c1973338569f6d0ed33b0e44f1a8511e which is post-2.99.42 and not released yet. It's not a code patch, it's a systemd unit file patch. [12:55:43] so you can edit the unit file to fix it. [12:55:49] before I rm'd it, it was drwx------ 2 gdnsd gdnsd 100 Sep 11 15:45 /var/run/gdnsd/ [12:55:53] ah [12:56:31] because of , it's best if systemd manages the creation of the directory with those directives [12:57:11] yeah that got it [13:02:42] 10Traffic, 10Community-Tech, 10MediaWiki-Parser, 10Operations: Show SVGs in wiki language if available - https://phabricator.wikimedia.org/T205040 (10Niharika) [13:04:49] 10Traffic, 10Community-Tech, 10MediaWiki-Parser, 10Operations: Show SVGs in wiki language if available - https://phabricator.wikimedia.org/T205040 (10Niharika) @MaxSem @Mooeypoo If this ticket is a lot of work (sounds like it), it might be better to split it in smaller chunks. [13:36:42] (next 2.4.x httpd release will have TLS 1.3 support) [14:45:32] bblack: i can be ready in like 15min [14:54:17] ok [15:00:58] bblack: alright, ready to commit the changes, let me know when good to go (route 91.198.174.239/32 to 91.198.174.114 instead of 91.198.174.121) [15:14:46] XioNoX: ready whenever! [15:16:52] bblack: done and seems to be working fine [15:16:56] seems to be working fine, yeah [15:19:51] thanks! [15:48:49] 10Traffic, 10Fundraising-Backlog, 10Operations, 10fundraising-tech-ops: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (10CCogdill_WMF) [16:06:38] bblack: dunno when you're planning on rolling it back, but I'm going to be biking to ulsfo in ~1h, so ~30min unavailable [16:18:56] XioNoX: I'm not planning to roll it back [16:21:57] ah! Cool then :) Will update v6 then too if it's not temporary [16:36:17] yes, please do! [16:39:05] done [16:39:19] thanks! [17:48:46] 10Traffic, 10DNS, 10Operations, 10WMF-Communications, and 4 others: Move Foundation Wiki to new URL when new Wikimedia Foundation website launches - https://phabricator.wikimedia.org/T188776 (10bd808)