[08:40:30] 10Traffic, 10Varnish, 10Operations, 10Maps (Tilerator), and 2 others: Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776 (10Mholloway) This needs some work: - We need to document that this should be disabled on initial data import / tile generation (or, better yet, enforce tha... [08:42:21] 10Traffic, 10Varnish, 10Operations, 10Maps (Tilerator), and 2 others: Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776 (10Mholloway) Blocking this until the service/config is up to date on all hosts. [08:43:14] 10Traffic, 10Varnish, 10Operations, 10Maps (Tilerator), 10Reading-Infrastructure-Team-Backlog (Kanban): Tilerator should purge Varnish cache - https://phabricator.wikimedia.org/T109776 (10Mholloway) [08:57:52] vgutierrez, hi, so are you happy with https://gerrit.wikimedia.org/r/#/c/operations/software/certcentral/+/459785/ now? [08:58:16] yup [08:58:36] didn't +2 it yet, cause I didn't know if you were going to include the signal integration test or not [08:59:37] oh right [08:59:53] I've got a couple hours in about an hour so I'll have a go at that [09:00:03] ok [09:03:47] actually it's easy, uploaded [09:04:14] :) checking [09:04:58] 10Traffic, 10Community-Tech, 10MediaWiki-Parser, 10Operations, and 2 others: Show SVGs in page language if available - https://phabricator.wikimedia.org/T205040 (10fgiunchedi) Adding #thumbor too since I'm sure it'll be affected as well. re: swift space concerns I don't think it'll be a problem unless the... [09:06:31] hmmm so you're not triggering a SIGHUP, just invoking the function passed as a second argument to signal.signal() [09:08:07] yeah [09:08:33] don't think we really need to test the basic functionality of python's signal module do we? [09:09:01] nope [09:09:16] but how are you checking there that the function passed as the second parameter is the expected one? [09:10:54] that's tricky to do [09:11:08] the API could change to using some other function that calls CertCentralConfig.load and we wouldn't notice [09:13:00] not sure if there's any way we could detect that [09:15:21] well.. you're testing that load() is being used [09:15:36] hmm maybe you could test for the log message as well? [09:15:46] guess we could check the base_path too? [09:18:52] you mean the parameters passed to load()? [09:19:54] yeah [09:20:35] yup [09:21:41] I don't think we should really test for specific log messages [09:26:12] vgutierrez, pushed one that checks what CertCentralConfig.load gets called with [09:26:34] cool [09:31:57] wanna merge the 4 things on top of it? :) [09:32:39] yup [09:35:48] 10Traffic, 10Maps, 10Operations, 10Reading-Infrastructure-Team-Backlog (Kanban): Decide on Cache-Control headers for map tiles - https://phabricator.wikimedia.org/T186732 (10Gehel) While we validated with @BBlack that the expected invalidation load on varnish is reasonable, we have not checked that this lo... [09:46:27] ok [10:08:26] vgutierrez, so remaining stuff.. you were going to deal with the subject changes commit, and we need to get the packaging stuff done [10:08:37] and there's some misc stuff around [10:09:05] like 3 commits to review/fix and the VM request [10:09:55] yup, I'm working on the subject changes commit [10:10:35] the packaging stuff CR, it would be good to get it reviewed by someone with more debian pkg expertise than me [10:10:39] dunno if ema can take a look [10:20:59] sure [10:21:29] https://gerrit.wikimedia.org/r/#/c/operations/software/certcentral/+/458554/ [10:21:33] <3 [10:24:20] Krenair: right now the code is taking certcentraltest.beta.wmflabs.org as something different to certcentraltest.BETA.wmflabs.org, according to the tests [10:24:51] "Certificate test_certificate type ec-prime256v1 has CN certcentraltest.BETA.wmflabs.org but is configured for certcentraltest.beta.wmflabs.org, moving back to re-issue" [10:25:26] I'll fix that as well [10:25:28] ok [10:25:58] wonder if there's other normalisation that we're missing vgutierrez [10:40:02] oh and I think the puppetisation needs to be done before the VM request [10:40:29] Krenair: nice, there are a bunch of lintian errors now that we want to fix https://integration.wikimedia.org/ci/job/debian-glue-non-voting/2300/console [10:41:22] hm [10:41:29] I don't remember seeing those lintian errors when building it [10:46:20] Krenair: on my laptop I get the following [10:46:22] E: certcentral source: missing-build-dependency-for-dh_-command dh_python2 => dh-python [10:46:25] E: certcentral source: missing-build-dependency-for-dh-addon python3 => dh-python [10:46:28] W: certcentral source: missing-license-paragraph-in-dep5-copyright gpl-3+ (paragraph at line 11) [10:46:31] W: certcentral source: ancient-standards-version 3.9.5 (released 2013-10-28) (current is 4.2.1) [10:50:37] I'll add comments to the CR for those [11:09:30] Krenair: also if certcentral is a program rather than a module, the binary package should be called certcentral, not python3-certcentral [11:10:24] ok [11:11:07] see https://www.debian.org/doc/packaging-manuals/python-policy/ for the details [11:13:54] sweet, debian-glue-non-voting is pleased [11:14:42] Krenair: phab id missing in the commit log, after which the debianization work can be merged IMHO [11:16:01] ema, amended commit message to include the ticket [11:18:36] my laptop is about to die [11:18:54] uh [11:18:57] poor laptop [11:19:24] now that the binary package has been renamed we also need to `mv debian/python3-certcentral.manpages debian/certcentral.manpages` [11:24:28] mmh also tests don't seem to be discovered [11:25:02] I: pybuild base:217: cd /build/certcentral-0.1/.pybuild/cpython3_3.7_certcentral/build; python3.7 -m unittest discover -v [11:25:05] ---------------------------------------------------------------------- [11:25:08] Ran 0 tests in 0.000s [11:30:25] oops looks like we made the same change :) [11:30:56] ouch, sorry! I thought I'd go ahead while you were bringing your laptop back to life [11:31:41] yeah np [11:31:47] I just had to go and find power [11:31:52] Krenair: how do you run the tests? [11:32:11] tox -e unittest [11:32:23] or `tox` for that plus pylint/flake8 etc [11:32:31] jenkins does it successfully [11:32:57] wonder what we need to do to make it pick up the tests [11:33:35] Krenair: it might be as easy as adding some magic variable to debian/rules [11:33:39] export PYBUILD_TEST_TOX=1 [11:33:42] or something [11:34:33] ok [11:34:38] probably adding python3-dev to build-depends too [11:36:32] heh, and tox :) [11:37:06] trying on my machine [11:48:52] Krenair: actually running tox when building the package seems like a bad idea, let's add that to the jenkins definition later? [11:55:31] FYI, https://phabricator.wikimedia.org/T136562 [11:55:51] lvs2009/2010 have two disks but OS is installed on only the first, they don't have raid configured [11:56:02] ema, okay [11:56:19] ema, want to remove the debian/rules export line I just added? [11:56:57] Krenair: yup :) [11:58:52] Krenair: jenkins already does run the tests with certcentral-tox-docker, so \o/ [12:03:54] ema, alright so is it ready? [12:05:30] Krenair: remove the PYBUILD_TEST_TOX line and then yes, ready to merge [12:06:54] done [12:08:19] ema, vgutierrez ^^ lvs2009/10 likely need a rebuild [12:19:58] paravoid: ack [12:20:14] paravoid: did you open a task for that? [12:20:27] oh.. got it [12:20:32] not a separate one, up to you [12:20:35] there's this mega-task [12:32:11] 10Traffic, 10Operations: lvs2009/lvs2010 with no RAID configured - https://phabricator.wikimedia.org/T205970 (10Vgutierrez) p:05Triage>03High [12:34:02] lvs100[7-9]|lvs101[012]|lvs2*) echo partman/flat.cfg ;; \ [12:34:03] lvs100[1-6]|lvs101[3-6]|lvs[345]*|lvs2009|lvs2010) echo partman/raid1-lvm.cfg ;; \ [12:34:12] I guess that's the culprit [12:57:48] indeed [13:27:19] 10Traffic, 10Operations: lvs2009/lvs2010 with no RAID configured - https://phabricator.wikimedia.org/T205970 (10Vgutierrez) Looks like the culprit is https://github.com/wikimedia/puppet/blob/production/modules/install_server/files/autoinstall/netboot.cfg#L126-L127: ``` lvs100[7-9]|lvs101[012]|lvs2*) ec... [13:31:13] ema, thanks [13:31:51] yw! [15:57:35] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Dzahn) @Papaul please see T205970 [15:58:15] 10Traffic, 10Operations: lvs2009/lvs2010 with no RAID configured - https://phabricator.wikimedia.org/T205970 (10Dzahn) i would say this is part of T196560 which is still open. i left a comment there. [15:58:28] speaking of LVS, is there a task for the replacement of lvs1001-06? [16:04:54] there's a bunch of related tasks, but I think the overall task state of lvs1001-6 -> lvs1007-12 -> lvs1013-16 may still be a bit messy [16:06:17] the intent was to quickly get 13-16 in place and then clean it all up afterwards, but then 13-16 are blocked on the eqiad switch work... [16:06:26] (last I recall, for further progress) [16:07:47] 10Traffic, 10Operations: lvs2009/lvs2010 with no RAID configured - https://phabricator.wikimedia.org/T205970 (10Dzahn) [16:07:51] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10Dzahn) [16:12:39] gah [16:16:00] 10netops, 10Operations: Renumber office-DC interconnect link - https://phabricator.wikimedia.org/T205985 (10ayounsi) p:05Triage>03High [16:22:19] 10netops, 10Operations, 10Patch-For-Review: Renumber office-DC interconnect link - https://phabricator.wikimedia.org/T205985 (10ayounsi) [16:44:16] 10Traffic, 10Operations: Simplify comment misc-frontend.inc.vcl.erb - https://phabricator.wikimedia.org/T205988 (10Imarlier) [16:50:00] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Grow frack-administration-codfw to /28 - https://phabricator.wikimedia.org/T204271 (10ayounsi) [17:14:03] 10Traffic, 10Operations: lvs2009/lvs2010 with no RAID configured - https://phabricator.wikimedia.org/T205970 (10Papaul) @Vgutierrez does this mean reinstalling both LVS servers (lvs2009 and lvs2010) if yes please elaborate how you want to approach this . Thanks. [17:24:40] 10Traffic, 10Operations, 10decommission, 10ops-ulsfo, 10Patch-For-Review: decommission lvs400[1-4].ulsfo.wmnet - https://phabricator.wikimedia.org/T178535 (10ops-monitoring-bot) wmf-decommission-host was executed by robh for lvs4001.ulsfo.wmnet and performed the following actions: - Revoked Puppet certif... [17:24:52] 10Traffic, 10Operations: Simplify comment misc-frontend.inc.vcl.erb - https://phabricator.wikimedia.org/T205988 (10herron) p:05Triage>03Normal [17:24:57] 10Traffic, 10Operations, 10decommission, 10ops-ulsfo, 10Patch-For-Review: decommission lvs400[1-4].ulsfo.wmnet - https://phabricator.wikimedia.org/T178535 (10ops-monitoring-bot) wmf-decommission-host was executed by robh for lvs4002.ulsfo.wmnet and performed the following actions: - Revoked Puppet certif... [17:25:07] 10Traffic, 10Operations, 10decommission, 10ops-ulsfo, 10Patch-For-Review: decommission lvs400[1-4].ulsfo.wmnet - https://phabricator.wikimedia.org/T178535 (10ops-monitoring-bot) wmf-decommission-host was executed by robh for lvs4003.ulsfo.wmnet and performed the following actions: - Revoked Puppet certif... [17:25:19] 10Traffic, 10Operations, 10decommission, 10ops-ulsfo, 10Patch-For-Review: decommission lvs400[1-4].ulsfo.wmnet - https://phabricator.wikimedia.org/T178535 (10ops-monitoring-bot) wmf-decommission-host was executed by robh for lvs4004.ulsfo.wmnet and performed the following actions: - Revoked Puppet certif... [17:26:23] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Grow frack-administration-codfw to /28 - https://phabricator.wikimedia.org/T204271 (10ayounsi) [17:27:02] 10netops, 10Operations: IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 noisy alert - https://phabricator.wikimedia.org/T205829 (10herron) p:05Triage>03High [17:27:17] bblack: hi! if you have time I'd have a question - is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/463957/ something that I can merge/deploy tomorrow by myself or better to wait for somebody from traffic? (I am trying to schedule a maintenance window for the announcement) [17:27:47] 10Traffic, 10Operations, 10decommission, 10ops-ulsfo: decommission lvs400[1-4].ulsfo.wmnet - https://phabricator.wikimedia.org/T178535 (10RobH) [17:28:34] 10Traffic, 10Operations, 10decommission, 10ops-ulsfo: decommission lvs400[1-4].ulsfo.wmnet - https://phabricator.wikimedia.org/T178535 (10RobH) So, these are all racked in the two new racks, but without any power or network. As such, I'll just continue with the remainder of the steps (puppet was never dis... [17:28:41] 10Wikimedia-Apache-configuration, 10Operations, 10Wikimedia-Site-requests: Temporarily redirect sgs.wikipedia.org to bat-smg.wikipedia.org until bat-smg->sgs move can be done - https://phabricator.wikimedia.org/T204830 (10herron) p:05Triage>03Normal [17:32:49] elukey: probably better to wait for ema just in case (or me), although it's hypothetically pretty safe. I'm also not 100% sure about defining the new backend and using it all in one commit... maybe that works, or maybe they should be split up... [17:33:04] (looking) [17:38:40] elukey: honestly since I don't remember now, it's hard to say for sure just looking at the code. In any case it wouldn't hurt to split them (define just the app_directors part for matomo1001 first without referencing it in piwik. That part can be deployed independently ahead and doesn't change anything user-facing) [17:39:03] on the trafficserver side it kinda doesn't matter since it's all non-live testing and in fluex [17:39:06] *flux [17:40:02] bblack: ack! Will split and wait for ema tomorrow :) [17:40:04] thanks! [17:40:05] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Grow frack-administration-codfw to /28 - https://phabricator.wikimedia.org/T204271 (10ayounsi) [18:12:11] 10Traffic, 10Operations: Simplify comment misc-frontend.inc.vcl.erb - https://phabricator.wikimedia.org/T205988 (10BBlack) Probably we need to do more than simplify the comment here, and instead actually fix/refactor the logic so it can work sanely. Either way, we'll need some relatively-bulletproof way to li... [18:49:02] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Grow frack-administration-codfw to /28 - https://phabricator.wikimedia.org/T204271 (10ayounsi) [21:41:52] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Grow frack-administration-codfw to /28 - https://phabricator.wikimedia.org/T204271 (10ayounsi) [21:43:34] 10netops, 10Operations, 10fundraising-tech-ops, 10Patch-For-Review: Grow frack-administration-codfw to /28 - https://phabricator.wikimedia.org/T204271 (10ayounsi) 05Open>03Resolved An oversight prevented frbast2001 to reach eqiad: codfw only advertised 10.195.0.0/25 to eqiad over ipsec. Making it a /24... [22:09:36] 10Traffic, 10Analytics, 10Operations, 10Services (blocked): Add Accept header to webrequest logs - https://phabricator.wikimedia.org/T170606 (10Pchelolo) We did enable the feature after all by looking at requests reaching #RESTBase, but that's not very convenient. Technically this is no more required. How... [23:51:59] 10Traffic, 10Community-Tech, 10MediaWiki-Parser, 10Operations, and 2 others: Show SVGs in page language if available - https://phabricator.wikimedia.org/T205040 (10Niharika) [23:53:25] 10Traffic, 10Community-Tech, 10MediaWiki-Parser, 10Operations, and 3 others: Show SVGs in page language if available - https://phabricator.wikimedia.org/T205040 (10Niharika)