[03:46:57] 10Traffic, 10Operations, 10Community-Liaisons (Oct-Dec 2017), 10Patch-For-Review, 10User-Johan: Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members - https://phabricator.wikimedia.org/T163251#3656120 (10Johan) [04:16:46] 10Traffic, 10Operations, 10Community-Liaisons (Oct-Dec 2017), 10Patch-For-Review, 10User-Johan: Communicate dropping IE8-on-XP support (a security change) to affected editors and other community members - https://phabricator.wikimedia.org/T163251#3656134 (10Johan) There will be a new reminder in [[ https... [07:37:24] hello people [07:37:29] on baham the last apt-get autoremove removed python-requests:amd64 (2.12.3-1), and prometheus-gdnsd-stats is not happy :) [07:38:36] (cron spamming root@) [08:04:41] <_joe_> elukey: how come it removed python-requests? [08:04:45] <_joe_> that's pretty unusual [08:05:14] _joe_ I didn't check but maybe it is not listed in the Core was generated by `/usr/bin/php /srv/mediawiki/multiversion/MWScript.php maintenance/getConfigurat'. [08:05:19] argh [08:05:21] paste fail [08:05:34] I meant to say: in the prometheus-gdnsd-stats deps ? [08:05:56] (ahahha I kinda segfaulted, nice) [08:06:00] <_joe_> that's not a debian package [08:06:32] ahhh it is in puppet right [08:06:37] this might explain why [08:08:54] should we just add python-request to the requirements of prometheus::node_gdnsd ? [08:11:55] (https://grafana.wikimedia.org/dashboard/db/dns for codfw doesn't look great) [08:13:33] ah good catch [08:15:53] elukey: fixing, thanks :) [08:16:24] :) [08:20:39] elukey, godog: https://gerrit.wikimedia.org/r/382129 [08:21:19] yup, lgtm [08:22:11] same from me :) [08:22:34] cool, merging [13:06:32] there's a ton of python modules in the autoremove list on a lot of our hosts, I was kinda wondering if some packages lost deps or something [13:06:51] the autoremove list on cp1065 was: [13:06:53] libnfsidmap2 libpgm-5.1-0 libsodium13 libtirpc1 libuuid-perl libzmq3 python-dateutil python-jinja2 python-m2crypto python-markupsafe python-msgpack python-requests python-zmq [13:07:31] (I did the autoremove on 1-2 caches, but not all yet) [13:10:29] they seems to me old dependencies of salt-minion [13:10:46] and salt-common [13:11:07] all the python-* that is, and I guess the lib* are dependencies of the others [13:12:34] bblack: ^^^ [13:32:29] yeah, python-zmq and python-msgpack were pulled in via salt-minion (plus a few deps of those like sodium) [13:33:13] it's likely that some declarations on those packages were omitted in the past since they were around due to salt already [13:33:39] libnfsidmap2 and libtirpc1 are obsolete due to the nfs-common/rpcbind removal a while ago [13:36:41] so zerofetch uses python-requests [13:36:46] I think python-requests has some problematic edge cases [13:36:57] yeah, so do some pybal scripts, but it's autoremove on LVSes too [13:37:45] mmh wait but in zero_update.pp we explicitly install it [13:38:05] what's really awful if you're of the mind to be super-tidy about such things, is that moving a package dep up to a puppet require_package leaves a hole later in the process [13:38:07] same for pybal::monitoring [13:38:24] un-applying that puppet require_package won't result in an autoremove, just a stale package installed explicitly for a no-longer-good reason [13:38:50] ema: right, but probably it got pulled in for package deps of other things before puppet got to attempt the explicit install [13:38:58] indeed [13:39:00] so now autoremove will kill it, but the next puppet run will add it back [13:39:25] (maybe when puppet is required to install a package, it should mark existing ones as explicitly installed :P) [13:39:57] eheheh indeed! [13:41:23] yeah so cp1065 that I did the autoremove on, it failed the zerofetch cron 2x in a row, then the next puppet run hit and reinstalled python-requests [13:43:28] so given lvs, caches, dns all affected by the same basic cycle, I guess I'll do "apt-get autoremove; apt-get install python-requests" as I get to that part on them [13:44:00] should we add apt-get autoremove to the puppet-run script after apt-get update and before the puppet run? [13:48:07] it would be nice if it were already in place since a long time ago [13:48:17] I would hate to be the person who pushes that change out right now though :) [13:51:12] volans: but we should move prometheus-gdnsd-stats to a deb package first right? Otherwise we'd loose python-requests every time, getting it back during the puppet run (so chance to loose metrics in the timeframe) [13:52:32] bblack: eheheh, ofc I think we should first do a manual pass, canaries first, of autoremove + puppet run [13:52:43] elukey: prometheus-gdnsd-stats runs on a crontab and output to file or is a daemon? [13:53:49] volans: there is definitely a cron that does this /usr/local/bin/prometheus-gdnsd-stats --outfile /var/lib/prometheus/node.d/gdnsd.prom [13:54:19] looks like a cron [13:55:12] where is the requests requirement? require_package('python-prometheus-client') [13:56:33] and at most we might loose one minute of metrics I guess, [13:57:59] volans: I think you have your git repo not updated, em*a fixed it this morning [13:58:35] might be [13:58:47] not working on puppet repo atm [13:59:02] yep, is fixed [14:00:41] my point is that we'd need to find and change all the corner cases before making the automatic autoremove move [14:03:04] elukey: absolutely agree, I think we need first a manual pass, cluster by cluster, starting with the canaries for each cluster [14:07:25] <_joe_> I am very against having an automatic autoremove [14:07:44] <_joe_> autoremove is a dumb/blind process, as just shown by this episode [14:08:00] _joe_: indeed, but uncovered an issue in out puppettization [14:08:08] <_joe_> also, you're OT here [14:08:15] <_joe_> ;) [14:08:20] and was working by chance, on a new installed/reimaged host would have failed ;) [14:10:05] cronspam is one of our best alarming systems [14:10:16] * elukey runs away from Riccardo [14:12:21] lol [14:14:28] yeah, let's not add automatic autoremove, this is bound to break all kinds of setups for virtually no gain [14:16:25] the only reason autoremove came up is because I realized it's been a while since we've done a general "apt-get upgrade" sort of thing on the various traffic hosts, and they're behind on basic packages [14:16:56] so I was upgrading those (while carefully trying to avoid experimental/in-progress upgrades like pybal+nginx), and noticed the long autoremove list and started trying it on some of the hosts [14:38:25] 10Traffic, 10Operations, 10ops-esams: esams rack OE10 power redundancy issues? (cp3030-9) - https://phabricator.wikimedia.org/T177403#3657409 (10BBlack) [14:48:32] https://grafana.wikimedia.org/dashboard/db/varnish-machine-stats?orgId=1&var-server=cp4025&var-datasource=ulsfo%20prometheus%2Fops&from=now-6h&to=now [14:49:05] ema: can you peek at cp4025 above? something odd there. fast-rising mailbox lag on the backend, but also an increase in exp lockops on the frontend, possible transient memory spike? [14:49:34] maybe something complicated like excess real frontend traffic driving up transient allocation, pushing out buffercache space for backend disk, thus causing disk-slowness mailbox lag? [14:50:55] I lean more and more towards eventually splitting fe+be machines physically, during some future hw iteration [14:51:16] (and making them smaller/simple machines with single cpu dies / numa domains) [14:51:41] interesting, fetches haven't started failing there [14:52:03] backend connections piled up to ~2.2k [14:52:04] I only noticed because mbox lag hit warning state in icinga while I was there [14:52:06] and then it recovered [14:53:14] tho mailbox lag hasn't recovered yet [14:53:27] with a hardware split they don't have to be symmetrical, either [14:53:59] we could do something like 5x text-fe + 7x upload-fe + 9x combined-be or whatever [14:55:32] (but then again, that's all in a varnish-only world that all this thinking makes sense. We may find in an ATS world we end up layering things differently? running fe+be as a single daemon while still getting the numerical benefits of the current setup?) [14:58:22] uh [14:58:27] root@cp4025:~# varnish-backend-restart [14:58:30] cp4025.ulsfo.wmnet: pooled changed yes => no [14:58:30] Job for varnish.service canceled. [14:59:26] concurrent agent run [14:59:34] "run-no-puppet varnish-backend-restart" [15:00:23] :) [15:00:30] * ema gets reminded to use the shit he wrote [15:01:20] I think agent did restart it, just needs repool [15:01:37] y [15:02:18] in other news, I've uploaded the latest varnish-modules upstream to debian [15:02:46] we can proceed uploading XioNoX's awesome v5 packages to apt.w.o (experimental) and then see what breaks! [15:03:37] 10Traffic, 10Operations, 10ops-esams: esams rack OE10 power redundancy issues? (cp3030-9) - https://phabricator.wikimedia.org/T177403#3657475 (10faidon) [15:04:30] yay! [15:05:57] XioNoX: want to push your changes to operations/debs/varnish4 [15:06:00] ? [15:06:29] 10Traffic, 10DC-Ops, 10Operations, 10ops-esams: Multiple systems in esams OE10 showing PSU failures - https://phabricator.wikimedia.org/T177228#3657486 (10BBlack) [15:06:32] (project name not particularly future proof clearly) [15:07:20] :) [15:07:46] probably should if we're upload to apt/experimental, so we have some record of the git stuff to work from going forward [15:07:51] (or look at to diagnose issues) [15:08:38] bblack: you mean push the git changes to the repo? [15:08:50] ema: sure, is there a doc for that? [15:09:09] git push --all -f --do-it [15:09:12] something like this [15:10:45] normally we'd follow the usual gerrit workflow, but for a new upstream it doesn't really make sense [15:11:10] as in we're not gonna review the diff between 4.1.3 and 5.1.3, are we? [15:11:41] right [15:11:56] so the way I handle this in the nginx case, is push a new baseline upstream branch w/o gerrit [15:12:09] then our debianization/patching/release commits w/ gerrit on a different branch, usually [15:12:34] I don't know what branch structure XioNoX currently has in that work, though [15:13:05] same as we used for the v4 packages: upstream/pristine-tar/debian-wmf [15:13:12] "git push -u origin basebranchname" to send the upstream base branch w/o review [15:13:18] (from a checkout of that branch) [15:13:20] I need to find where the repo is :) [15:13:37] then the more-usual "git push origin HEAD:refs/for/whatever" for the debianization/patches/release stuff of ours [15:13:43] XioNoX: ayounsi@traffic-varnish5:~/varnish4 [15:14:06] (yes I've been spying on you) [15:14:46] XioNoX: don't forget to classify the git origin first! https://git-man-page-generator.lokaltog.net/#3e5163f5c1696f3d904f91a2252b8413 [15:16:54] pristine-tar 1eadaf6 [origin/pristine-tar: ahead 1] pristine-tar data for varnish_5.1.3.orig.tar.gz [15:16:57] upstream 28a9321 [origin/upstream: ahead 1] Imported Upstream version 5.1.3 [15:17:00] debian-wmf c9ef443 [origin/debian-wmf: ahead 4] Update changelog for Varnish 5.1.3 [15:17:38] haha I'm lost [15:18:13] (sorry that last one is a joke!) [15:18:18] ema: I don't think it's on ayounsi@traffic-varnish5:~/varnish4 , but instead on ayounsi@copper:~/varnish4 [15:18:32] Your branch is ahead of 'origin/debian-wmf' by 6 commits. [15:18:46] XioNoX: oh ok! [15:19:29] ah cool, a new commit to update libvarnishapi1's symbols [15:19:30] nice [15:20:07] any chance that we could try to compile varnishkafka with the new libs just to figure out it anything explodes? :D [15:20:20] XioNoX: git checkout upstream ; git push -u origin upstream [15:20:34] XioNoX: same for pristine-tar [15:21:20] XioNoX: and then git checkout debian-wmf ; git push origin HEAD:refs/for/debian-wmf/varnish5-upgrade [15:21:38] elukey: sure :) [15:22:20] of course, I need to scp everything to my laptop, I can't just push it from copper [15:24:01] bblack: lol git-classify-origin looks painfully legit [15:24:34] it all does [15:24:53] haha nice [15:25:04] there's really no way for anyone to know whether it's a joke if someone says something like "Don't forget to git-randomize-classifiers first!" or whatever without context [15:25:20] because nobody can ever memorize all the insane ductwork of commands and options that is git [15:25:59] maybe it's just me, but every time I run 'man git-blah' I already start shaking my head before pressing enter [15:27:09] just remember, git isn't really a version-control system. it's a filesystem that remembers all the history of everything done to it, and the "git" command is like some low-level kernel filesystem API that has nothing to do with POSIX. [15:27:21] trying to use it as a high-level version control tool is all on you, not the authors :) [15:27:22] fyi, the link between ulsfo and eqord is down, Telia is working on it [15:37:55] 10Traffic, 10Operations, 10Performance-Team (Radar): Upgrade to Varnish 5 - https://phabricator.wikimedia.org/T168529#3657709 (10ema) So, packages-wise, I've packaged and uploaded [[ https://packages.qa.debian.org/v/varnish-modules/news/20171004T150923Z.html | varnish-modules 0.12.1 ]] to Debian unstable. @a... [15:40:57] !log acamar (recdns@codfw) - base package upgrades [15:41:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:42:05] so the cold symptoms are still pestering me, going to bed now [15:42:10] o/ [15:42:47] get better! :) [15:43:11] yes, don't contaminate us please [15:46:15] I'm trying to do `git push origin HEAD:refs/for/debian-wmf/varnish5-upgrade` but gerrit complains about no change-id [15:46:56] I ammended the last commit to include one, but any idea how to add one for all the previous commits (not only the most recent one) [15:47:21] upstream and pristine-tar have been pushed [15:48:17] XioNoX: you'll have to amend all the new commits [15:48:28] what does "git status" show on your varnish5-upgrade branch? [15:48:35] err, debian-wmf branch, sorry [15:48:52] Your branch is ahead of 'origin/debian-wmf' by 6 commits. [15:48:57] ok [15:49:03] so since the origin is right, etc... [15:49:18] have you used interactive rebase before? [15:49:27] "git rebase -i" [15:49:54] yes, but not enough time to remember how to properly use it [15:50:01] if you type that, it's going to bring up a vim (or whatever) editor with a list of your 6 commits [15:50:16] indeed [15:50:17] change the first column to "r" for "reword" on all 6 commits and save. [15:50:36] then it will step through commitmsg-ammending vims for all 6 commits (just save them as is) [15:50:43] and that will invoke the hooks to add changeids to them all [15:51:21] Super cool! [15:51:34] bblack, [15:51:35] https://gerrit.wikimedia.org/r/382174 Imported Upstream version 5.1.3 [15:51:35] https://gerrit.wikimedia.org/r/382175 Adapt WMF specific patches for Varnish5 [15:51:35] https://gerrit.wikimedia.org/r/382176 Update changelog for Varnish 5.1.3 [15:51:35] https://gerrit.wikimedia.org/r/382177 Varnish5: Install devicedetect.vcl in the proper path [15:51:35] https://gerrit.wikimedia.org/r/382178 Varnish5: Update libvarnishapi1.symbols [15:57:06] yeah so the debian-glue checks in jenkins don't like it [15:57:18] I think that git review would have done that automatically ;) [15:57:21] I could sort of understand that on the first commit given how it's structured [15:58:24] 10Traffic, 10Operations, 10Performance-Team (Radar): Upgrade to Varnish 5 - https://phabricator.wikimedia.org/T168529#3657787 (10ayounsi) pristine-tar and upstream pushed to origin debian-wmf pushed to gerrit: https://gerrit.wikimedia.org/r/382174 Imported Upstream version 5.1.3 https://gerrit.wikimedia.org... [15:58:47] but the next patch says "adapt WMF specific patches", yet the patches are failing to cleanly apply [15:59:04] near the bottom of: https://integration.wikimedia.org/ci/job/debian-glue/771/console [15:59:30] oh, it's using 4.1.8 as a baseline, because changelog isn't fixed up yet there [15:59:31] hmmm [15:59:52] anyways, don't merge those yet [16:00:11] (also, the changelog patch should probably be the last in the series, or else we've got git changes not recorded in the versioning, etc) [16:01:16] you can fix that with "git rebase -i" as well too, just re-order the lines and it will re-arrange the commit history [16:03:48] aside from the re-ordering, and aside from expecting the earlier commits to fail until we get to the changelog one at the end, the results here are probably still going to have to be dealt with: https://integration.wikimedia.org/ci/job/debian-glue/773/console [16:04:15] which is lintian failures a bit up from the bottom there, e.g. "varnish source: newer-standards-version 3.9.8 (current is 3.9.6)" [16:05:20] there's quite a few lintian issues shown there. some may be resolveable with changes to the packaging. others might have to be ignored in e.g. [16:05:23] debian/source.lintian-overrides [16:05:48] bblack: would it makes more send to squash all those commits? [16:06:35] and reordering will not mess with gerrit? (as they already have a review ID, etc.. [16:06:38] ) [16:06:40] probably not. can just live with -1 on the earlier commits so long as it's +1 at the end of the chain [16:07:02] (it's kind of silly when you think about it, that jenkins expects a perfect packaging state on each individual commit here, but whatever) [16:07:19] the reodering won't mess with gerrit, no [16:07:57] the re-ordering may fix a few of those lintian issues too, because I think the commits-after-changelog-update is what's causing the overlong/crazy .gbp123456 type of versioning [16:08:15] (it's trying to record in the version string that you're building from a git commit sha that's beyond the last changelog update) [16:10:21] Updated Changes: [16:10:21] https://gerrit.wikimedia.org/r/382176 Update changelog for Varnish 5.1.3 [16:10:21] https://gerrit.wikimedia.org/r/382177 Varnish5: Install devicedetect.vcl in the proper path [16:10:21] https://gerrit.wikimedia.org/r/382178 Varnish5: Update libvarnishapi1.symbols [16:10:54] I moved the changelog update to the end [16:13:23] hmmm still the gbp package naming and lintian failures related: https://integration.wikimedia.org/ci/job/debian-glue/777/console [16:14:10] 16:12:10 dpkg-deb: building package `varnish' in `../varnish_5.1.3-1wm1+0~20171004161018.777+jessie+wikimedia~1.gbp791d98_amd64.deb' [16:14:59] that is the right sha1 [16:15:14] maybe jenkins always uses the long versioning format like that [16:15:35] Train is arriving at San Jose, have to bike to the nanog venue and catch breakfast, ttyl! [16:16:00] ok [16:16:17] I'm running a jenkins recheck of this debian-glue stuff on ema's last 4.1.8 package to see how it differs [16:32:22] ok, so, apparently a bunch of those lintian issues are mere warnings ("W:" prefix), and are normal and usually don't cause issue [16:32:37] the one new one that's error-level ("E:" prefix) is the last one listed there: [16:32:42] 16:12:22 E: varnish: embedded-library usr/bin/varnishtest: zlib [16:33:10] so, the lintian checks think that varnishtest is static-linking zlib which is a no-no [16:33:19] no idea if that's true or a false-positive of some kind [16:35:57] https://gerrit.wikimedia.org/r/#/c/382174/1/bin/varnishtest/Makefile.am [16:36:18] ^ the change of the libvgz entry in LDADD there is suspicious (from .la to .a).... [17:00:15] https://github.com/varnishcache/varnish-cache/commit/932035b6d11896e41a9728e0c3fd809f01e0b56b [17:00:26] ^ this is the offending commit, which has been in upstream since release 5.1.0 [17:02:29] libvgz is a locally-modified copy of zlib anyways, I think it's basically unavoidable that upstream varnish builds in such a way that they're embedding zlib and need rebuilds for zlib sec fixes, etc [17:02:48] I think the changes above about static-linking libvgz just made the situation more-apparent to lintian [17:02:56] so we probably need some kind of lintian override for this [17:16:02] bblack: any pointers on how to do that? [17:17:38] bblack@alaxel:~/repos/varnish4$ cat debian/varnish.lintian-overrides [17:17:38] varnish: embedded-library usr/lib/*/varnish/libvgz.so: zlib [17:18:00] XioNoX: ^ that's the apparently already-existing one for this issue [17:18:20] in the varnish4 case, libvgz was shared, so the zlib-embedding override was applied to libvgz.so [17:18:40] I think now, we have to apply that same override to all the binaries that now statically link libvgz.a [17:19:32] so maybe change that line (in debian/varnish.lintian-overrides), to: [17:22:23] varnish: embedded-library usr/bin/*: zlib [17:22:25] varnish: embedded-library usr/sbin/*: zlib [17:22:30] (I think you need both lines) [17:23:52] bblack: how did you find those? [17:25:39] well, I knew we needed to override lintian. debian docs (or fuzzy past memory) says that goes in debian/.lintian-overrides [17:26:16] there's an existing file there for the varnish package making the same exception we want, in the current varnish4 packaging, just for the wrong target binar(y|ies) - libvgz.so instead of usr/bin/varnishtest and such [17:27:01] anyways, maybe make that little change as a separate commit and rebase it back before your changelog again and try another push [17:27:16] sure, thanks [17:28:58] New Changes: [17:28:59] https://gerrit.wikimedia.org/r/382193 Add lintian-overrides for statically linked lib [17:28:59] Updated Changes: [17:28:59] https://gerrit.wikimedia.org/r/382176 Update changelog for Varnish 5.1.3 [17:29:01] suspens! [17:31:24] yay, jenkins +2 ! [17:32:28] :) [17:32:47] e.ma probably could've sorted that out in like 1/10th the typing and time :) [17:32:56] (vs me, I mean) [17:37:45] haha [17:38:09] next step is to upload the packages to the experimental repo? [17:39:08] yeah, using reprepro [17:39:13] have you done that before here? [17:40:02] nop :) [17:41:12] ok, so basically first you need build outputs [17:41:36] e.g. start with a clean "build-area" when you build on copper (from the latest changesets that are now uploaded to gerrit) [17:42:05] (you can use your scp method to move changes there, or you can reset your checkout on copper to clean upstream state and use the anonymous pull commands from gerrit UI) [17:42:44] and then copy all of that "build-area" output directory at the end (all the .deb files as well as several others) to your local machine, then bounce it up to a directory in your homedir on install1002.wikimedia.org [17:42:49] https://wikitech.wikimedia.org/wiki/Reprepro [17:43:14] (for general reference) [17:43:21] then log into instal1002, "sudo -i" to root [17:43:29] cd ~ayounsi/where-you-dropped-all-those-files [17:43:59] reprepro -C experimental jessie-wikimedia includedsc *.dsc [17:44:04] reprepro -C experimental jessie-wikimedia include *.changes [17:45:44] let's see if they can compile again :) [18:00:08] 10Traffic, 10Operations, 10Wikidata, 10wikiba.se, 10Wikidata-Sprint-2016-11-08: [Task] move wikiba.se webhosting to wikimedia misc-cluster - https://phabricator.wikimedia.org/T99531#3658406 (10QChris) [18:05:33] bblack: see reprepro emails, seems like it worked [18:05:57] going to do the same for libvmod-netmapper [18:06:20] (which is already built against the new packaging of varnish5, too, right?) [18:06:35] yup [18:13:43] bblack: so there are two branches: master and debian. Should I push them directly or send them through gerrit? [18:14:07] they have respectively 3 and 6 commits [18:16:25] I thought it was the 3 branches talked about earlier? [18:16:47] pristine-tar, upstream, debian-wmf ? [18:20:00] not for libvmod-netmapper it seems [18:20:26] https://github.com/wikimedia/operations-software-varnish-libvmod-netmapper/ [18:24:10] oh, ok [18:24:21] I was still on the other repo in my head [18:25:02] well since we haven't seen the work at all, and "upstream" is in fact our own software [18:25:17] I'd say push the 3x master commits through gerrit for review first [18:25:46] and then let's review them there (in case needs amends or whatever), and then once they're merged through gerrit we can merge master back to debian-wmf and tack on the other 3, etc [18:31:13] New Changes: [18:31:13] https://gerrit.wikimedia.org/r/382207 Fix vmod_abi.h version parsing [18:31:13] https://gerrit.wikimedia.org/r/382208 Add description to vmod_netmapper.vcc [18:31:13] https://gerrit.wikimedia.org/r/382209 Bump version to 1.5 [18:58:24] XioNoX: yeah so, when you get a chance, +2 and merge those 3 changes on master first [18:59:12] and then when you go to push up the 6 on the debian branch, only the final 3 will generate new reviews in gerrit I think [18:59:33] but, it can also all wait until we upload packages to experimental and try it all out, like we're doing with the varnish-side changes [19:00:27] I'd say our normal flow would be to merge the changes before pushing the packages in both cases (varnish5 + libvmod-netmapper). But this is experimental-repo and likely we'll run into issues and have to amend $random_things. Not merging in gerrit could make initial fixups easier. [19:00:53] sounds good [21:09:36] 10Traffic, 10Operations, 10Patch-For-Review: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3658969 (10Aklapper) Two weeks later, is there more to do in this open task? Did https://gerrit.wikimedia.org/r/376751 "properly" fix this? Or is some "underlying issue investigation" needed? [21:37:13] I pushed the debian branch to gerrit so at least it's tracked somewhere (especially as copper is being decomm'ed) [22:23:44] alright, both varnish and libvmod-netmapper are now in experimental [22:27:38] 10Traffic, 10Operations, 10Performance-Team (Radar): Upgrade to Varnish 5 - https://phabricator.wikimedia.org/T168529#3659048 (10ayounsi) libvmod-netmapper pushed to gerrit as well. libvmod-netmapper / varnish pushed to the experimental apt repo. [22:46:23] 10Traffic, 10Operations, 10Patch-For-Review: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3659119 (10BBlack) There's still some overlap and/or confusion between the 503 issues in this ticket, T174932 and T145661, and there's some still lesser recurrent 503s in esams that we don't... [22:47:14] 10Traffic, 10Operations, 10Patch-For-Review: Text eqiad varnish 503 spikes - https://phabricator.wikimedia.org/T175803#3659123 (10BBlack) p:05High>03Normal