[02:51:12] 10Traffic, 10Operations, 10Wikimedia-Incident: upload.wikimedia.org returns HTTP status code 503 for truncated urls, not 404 - https://phabricator.wikimedia.org/T106517 (10Krinkle) [09:48:22] (for anybody interested, I am seeing some work in progress on the httpd 2.4.x branch to backport TLS 1.3 support) [09:50:01] nice [12:11:27] vgutierrez, hey, what do yo think about https://gerrit.wikimedia.org/r/#/c/operations/software/certcentral/+/456646/7/debian/source/format ? [12:11:30] you* [12:23:46] Also https://gerrit.wikimedia.org/r/#/c/integration/config/+/458156/ has given us a non-voting but failing job [12:24:12] failing because of our lack of upstream/0.1 tag in gerrit [13:46:02] Krenair: so.. I don't have an strong opinion about debian packaging [13:46:38] Krenair: on pybal we keep a separate branch for the releases but we keep the debian/ directory in master [13:47:14] so that allows us cherry-picking what we need for the next release from master [13:48:22] keeping versions as tag could work also [13:48:39] as long as we keep it documented in the README, I don't care honestly [13:50:36] we've our very own debian developer in traffic, maybe ema could spread some light here :) [14:00:44] Krenair: BTW.. from the same link that legoktm pasted in his comment [14:00:46] "(This section is only for source packages using the "1.0" format. For non-native packages using "3.0 (quilt)", the upstream debian/ directory is not an issue except taking a bit of extra space." [14:17:02] vgutierrez: one branch for upstream code, git tags for upstream releases, another branch for the debianization is a good approach IMHO [14:17:44] that's what we do already for varnish, ats, vhtcpd, ... [14:20:09] ema: in this case, the code is written by us, so "we are upstream" [14:21:03] you still have a virtual conception of upstream though [14:21:22] upstream is the software itself, and the debianization deals with packaging-level issues in a sep branch [14:21:32] right, that's the case for vhtcpd and others too. Still the distinction is useful I think, in particular given how git-buildpackage operates [14:21:39] ack [14:28:36] yeah, separate branches makes total sense [14:28:54] although i've often seen that debian prefers separate *repos* even in the case where you're the upstream [14:28:59] and i've not understood that yet [14:29:29] I still miss my travis CI instance building the packages using fpm [14:29:32] * vgutierrez hides [14:46:07] Krenair: ^^ so, I'd follow the approach ema suggested [14:53:27] fwiw I'm using a separate branch [14:53:45] vgutierrez, so, master is the 'upstream' branch [14:54:08] I can go over my workflow with cumin/debmonitor/spicerack, but later, I've a meeting in few minutes [14:54:10] then we tag stuff in there [14:54:27] and we have a separate branch containing just the debian directory, which doesn't exist on other branches? [14:54:33] yes [14:54:40] ema, ^ [14:55:05] Krenair: separate branch with all the upstream code plus the debian directory [14:55:18] ah right ofc, I meant that [14:55:30] sorry meeting, bbl [14:56:25] Krenair: see https://github.com/wikimedia/operations-software-varnish-vhtcpd and https://github.com/wikimedia/operations-software-varnish-vhtcpd/tree/debian [14:57:54] so [14:58:01] btw I would like to generalize the release script I have for cumin and make it a more general one, at least for python projects, stay tuned (if not E_TOO_MANY_GOALS) [14:58:43] master is the 'upstream' branch and contains the code but no debian/ directory [14:58:59] you make tags from there [14:59:14] which you merge into a debian branch which contains the debian directory and now also the code [14:59:15] ? [14:59:29] you rebase the debian branch with that tag I guess [14:59:46] is that something we can do with gerrit? [14:59:48] no need to merge it, as the debian branch won't change the code itself [15:00:08] it will only append the debian/ stuff [15:01:36] I don't understand [15:04:43] well you can't rebase for debian branch in the general, because once you've uploaded the debian branch for the first time, you can't rewind its history [15:05:08] so it ends up being a merge commit to pull the latest master onto the debian branch, then a local commit in the debian branch to update the changelog, then release deb from there [15:05:08] well.. push -f O:) [15:05:17] but yeah [15:05:20] that's saner [15:05:21] push -f breaks history for everyone else [15:07:31] No [15:07:35] No force-pushing. [15:08:16] I know I know, I was kidding [15:08:20] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/varnish/vhtcpd/+log/debian [15:08:31] vhtcpd is a good simple example, you can see the merge commits there in the debian branch history [15:14:08] okay so [15:14:27] master is the 'upstream' branch and contains code but no debian/ directory [15:14:37] you make tags from there [15:15:01] the debian branch is branched from master and you merge into it to make a new release, then commit to it to add a changelog entry [15:15:29] and from there you can build the package [15:15:32] yes? [15:26:44] correct [15:28:59] yup [15:37:01] so I need to split my packaging commit [15:37:03] master stuff setting up setup.py and co. [15:37:05] and a new commit adding the debian directory to the new branch afterwards [15:37:16] you can (should) do tags in both [15:39:17] 0.1 and 0.1-debian or 0.1 in both branches? [15:39:25] e.g. one such scheme that works well with other tooling, is the master branch has tag names like "v1.0.1" put on the release cuts there, and then the debian branch has tag names like "debian/1.0.1-1" which match the debian package naming [15:39:37] there can only be one target for one tag name across all branches [15:40:15] (v1.0.1 on master would match the final release commit there, and debian/1.0.1-1 would match the release commit there (the changelog update)) [15:41:01] is it me or is this seeming more complicated than it needs to be? [15:41:18] welcome to debian! [15:41:27] :| [15:41:28] (well really, no other distro is any better) [15:41:35] remember the goat thingie? :) [15:41:56] now it's when you kill it to get a valid .deb package [15:42:31] (now we will have to add a disclaimer: no goats were harmed during the development of certcentral) [15:42:40] the conventions here are built to scale (i.e. this works for major independent projects with a separate debianization maintainer, etc) [15:42:57] they do get to be a bit overkill when you're self-releasing like 1 text file [15:43:34] Krenair: if you think it's complicated, do not read gbp-buildpackage(1) [15:43:43] separation of debianization is a noble goal in general though. after all, someone else might want to package your software for CentOS or whatever. [15:43:59] (or FreeBSD, or OpenSolaris) [15:44:09] yup.. it's pretty nice if you think about it [15:44:14] I wouldn't mind them doing that on master [15:44:22] but yeah, it's painful at the beginning [15:45:11] integrations for N different OS installation/packaging issues in master gets cumbersome. Also, sometimes a target will need their own workaround patches that you can't easily upstream immediately. [15:46:01] e.g. debian might add a patch only in the debianized branch which is a quick fix to your source, that only applies to debian for whatever reason, but generalizing the fix as a commit to your master that doesn't break other OSes may require weeks of work and happen later. [15:47:22] the generalized idea is that as a software maintainer you develop reasonably-portable software, and as distribution maintainers various others package it for easy end-user consumption, and then there's a sometimes-delayed feedback loop where issues encountered by packagers filter back up to commits on your master branch that increase portability or fix bugs, etc [15:56:05] alright [15:59:38] Updated https://gerrit.wikimedia.org/r/#/c/operations/software/certcentral/+/456646/ to remove the debian stuff [16:01:10] when it's merged I will make the debian branch and upload that stuff for review there [16:01:40] vgutierrez, ^ [16:11:31] ack [16:12:33] Krenair: maybe that's worth to be mentioned in the commit message [16:13:36] ok [16:14:05] done [16:15:48] looking good, maybe let's wait for legoktm feedback before merging [16:17:02] vgutierrez, legoktm is likely very busy today [16:17:41] he's got a fixcopyright deployment window in 45 minutes [16:20:20] PS38.. I think it's the longest CR I've been involved since I've joined WMF [16:20:54] Half of those are me trying to appease Jenkins :P [16:21:48] and/or lintian [16:22:07] let's see if we can have a VM instance with the .deb deployed before Friday [16:22:16] I'll be out on vacations till the 25) [16:22:42] and it would be interesting to leave that done to let bblack play with the dns integration [16:23:11] is there a gdnsd instance anywhere in labs? [16:25:42] bblack ^^ [16:26:42] no idea [16:27:22] we do have jenkins running automated checks of commits to operations/dns using gdnsd, but I think that's not what you're looking for [16:27:52] in any case, even if there were a gdnsd instance in labs somewhere, it wouldn't yet have support for native acme challenge stuff [16:29:08] the commit for it is at least public, at https://github.com/blblack/gdnsd/commit/db7fff10b005b951890fa4ff7c843a1e37bbdc58 [16:30:10] so [16:30:15] but steps remain: finishing up a few other minor things I want in 3.x, some QA testing of some of the scarier 3.x bits against various other dns server implementaitons for interop (going on today), and then merging all the 3.x stuff back to the gdnsd/gdnsd master branch and cutting an alpha release we can try to package and deploy here [16:30:16] do we want to set up gdnsd in labs to try this out? [16:30:44] labs isn't a part of any of my normal workflows, generally speaking. [16:30:55] you may want to, once there's a release to even test! [16:31:24] the integration is pretty simple: a script that can ssh out commands to the authdns servers, and standard queries to a list of servers to confirm the challenge presence. [16:33:11] who is gonna write that script? [16:33:59] cercentral is already expecting that, so we need to swap the current /bin/echo with the actual script [16:34:29] I guess I can [16:34:49] there's some puppetization to go with it too, to use the authorized ssh keys, etc [17:04:01] 10netops, 10Operations, 10cloud-services-team: modify labs-hosts1-vlans for http load of installer kernel - https://phabricator.wikimedia.org/T190424 (10MoritzMuehlenhoff) Arzhel and I had a look and this doesn't seem to be ACL-related, the tftp packets are flowing in both directions. This is possibly a bug... [17:25:50] vgutierrez, Krenair: I'll look tonight [17:25:59] thanks [17:44:07] 10Traffic, 10Operations, 10ops-eqiad: cp1080 - kernel / bnxt_en failures - https://phabricator.wikimedia.org/T203194 (10Cmjohnson) @bblack @Dzahn I didn't get any direction with this but I assumed swapping the DAC is the appropriate attempted fix. Please let me know if this resolves the issue [18:32:32] 10Traffic, 10Operations, 10ops-eqiad: cp1080 - kernel / bnxt_en failures - https://phabricator.wikimedia.org/T203194 (10Dzahn) @Cmjohnson Thank you. I don't really know if that was the cause but looking at monitoring now everything is green. So i repooled the server just now. [18:36:16] 10netops, 10Operations, 10Wikimedia-Incident: asw2-a-eqiad FPC5 gets disconnected every 10 minutes - https://phabricator.wikimedia.org/T201145 (10ayounsi) asw2-a-eqiad members upgraded to 14.1X53-D47.3 (QFX host included). For the record, downtime was 10min. [18:42:04] 10Traffic, 10Operations, 10ops-eqiad: cp1080 - kernel / bnxt_en failures - https://phabricator.wikimedia.org/T203194 (10Dzahn) 05Open>03Resolved a:03Dzahn I guess we can close it and simply reopen it if it should happen again. Let me know if you think otherwise @ema @BBlack [20:05:11] 10Traffic, 10Operations, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: Sort out HTTP caching issues for fixcopyright wiki - https://phabricator.wikimedia.org/T203179 (10Legoktm) I didn't read the ULS code properly, it looks like we need `$wgULSAnonCanChangeLanguage` t... [20:37:18] 10netops, 10Operations, 10Wikimedia-Incident: asw2-a-eqiad FPC5 gets disconnected every 10 minutes - https://phabricator.wikimedia.org/T201145 (10ayounsi) Migration step toward a compatible topology. {F25670034} **Step 1** [] Disconnect already disabled links - fpc1-fpc3 (5m DAC) - fpc3-fpc5 (5m DAC) - fp... [21:09:37] 10Traffic, 10Operations, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: Sort out HTTP caching issues for fixcopyright wiki - https://phabricator.wikimedia.org/T203179 (10Bawolff) >>! In T203179#4558234, @Legoktm wrote: > Great :) And the special page TTL was fixed in h... [21:14:41] 10Traffic, 10Operations, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: Sort out HTTP caching issues for fixcopyright wiki - https://phabricator.wikimedia.org/T203179 (10Bawolff) > > Varnish seems to be caching it fine (Assuming you don't have cookies, which I imagin... [21:30:41] 10Traffic, 10Operations, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: Sort out HTTP caching issues for fixcopyright wiki - https://phabricator.wikimedia.org/T203179 (10BBlack) >>! In T203179#4561033, @Bawolff wrote: > Varnish seems to be caching it fine (Assuming you... [21:35:19] 10Traffic, 10Operations, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: Sort out HTTP caching issues for fixcopyright wiki - https://phabricator.wikimedia.org/T203179 (10Bawolff) However, if we use ULS to set the language, the assumption might be that people will touch... [22:45:00] 10Traffic, 10Operations, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Patch-For-Review: Sort out HTTP caching issues for fixcopyright wiki - https://phabricator.wikimedia.org/T203179 (10Legoktm) >>! In T203179#4561076, @BBlack wrote: >>>! In T203179#4561033, @Bawolff wrote: >> Varnish...