[07:31:39] 10Traffic, 10Operations, 10Patch-For-Review: Merge cache_misc into cache_text functionally - https://phabricator.wikimedia.org/T164609#4284629 (10ema) [07:54:54] elukey, paravoid: where is librdkafka1-0.11.3-1~bpo8+1+wikimedia1 source code hosted? [07:55:38] I'm not aware of a wikimedia1, so I wouldn't know :) [07:56:02] who's the last one in changes? [07:56:06] is not operations/debs/librdkafka ? :) [07:56:23] volans: I just created that [07:56:32] ahahah [07:56:38] * volans goes back to sleep [07:57:04] ah it was Andrew, for openssl 1.1 support [07:57:14] yep [07:57:16] * Rebuild for jessie-wikimedia backport. [07:57:16] * Use libssl 1.1 [07:57:43] so I don't think that we have any repo for it, it was probably a one off [07:57:55] hmmm [08:01:00] yeah, for those one-off builds we usually don't import them in git [08:01:16] mostly for the ones with longer maintenance (like openssl, nginx etc.( [08:04:42] yeah.. I can go from ~otto/librdkafka_0.11.3.orig.tar.gz and ~otto/librdkafka_0.11.3-1~bpo8+1+wikimedia1.debian.tar.xz on boron [08:06:42] or even simpler: "apt-get source librdkafka" on boron, that also guarantees that you're getting the exact source of what's in the repo [08:07:43] hmmm that's giving me the stretch one [08:08:09] yeah you need the jessie apt-src sources.list entry to get the jessie sources [08:08:55] try on pinkunicorn [08:09:40] yup [08:10:05] we've configured all apt sources on boron, if you need the old ones you can force that with "apt-get source foo=bar" [08:10:06] so... let's continue the 1 shot stuff or go the gerrit way? [08:10:41] see /etc/apt/sources.list.d/package-build-deb-src.list [08:11:13] given that this will be merged into the next release a one-off seems fine [08:11:26] gerrit would be more useful there was an ongoing diversion from upstream [08:11:43] +1 [08:11:45] nah.. luckily for us we got that merged on upstream [08:12:15] yep +1 for the one off [08:12:26] add yourself to changes so we can blame you when needed :P [08:12:28] so let's get rid of operations/debs/librdkafka [08:12:36] moritzm: mmh, what's the source package equivalent of apt-cache policy ? [08:13:14] I think the same policy as for the binary package, it's just a look up for the Source: value IIRC [08:13:45] apt-get source librdkafka=0.11.3-1~bpo8+1+wikimedia1 should work if the source has been imported to apt.wikimedia.org [08:13:58] but maybe Otto only used "includedeb" in reprepro [08:14:07] on boron we only have multiple sources.list entries for the apt-src lines, not the binary ones [08:14:22] so apt-cache policy librdkafka1 does not help much in figuring out the jessie version number [08:14:43] yeah, having all those binaries in apt source would be non-ideal :-) [08:14:55] sure, I'm not saying we should :) [08:15:27] just that I don't know how to list all available versions of a given source package [08:15:32] apt-get source librdkafka=0.11.3-1~bpo8+1+wikimedia1 actually works [08:16:05] ema: "reprepro ls foo" on install1002.wikimedia.org is the closest [08:16:23] right [08:16:31] it would be cool to have something like packages.debian.org, though [08:17:05] ha: apt-cache madison librdkafka [08:17:09] TIL [08:17:25] madison? lovely intuitive name [08:17:54] right, totally forgot about that [08:18:09] naming of that option is pretty stupid for a general command [08:18:33] vgutierrez: it refers to an internal Debian archive tool called madion [08:18:40] now replaced by something else [08:18:49] madison even [08:49:31] hmmm it looks like it's building the package.. and nothing it's on fire.. weird [09:29:28] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10Security: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4290110 (10Vgutierrez) p:05Lowest>03Normal a:03Cmjohnson [09:35:10] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293#4290145 (10Vgutierrez) [10:15:07] 10netops, 10Operations, 10fundraising-tech-ops: switch network port 2/0/3 (frdb1003) back to administration-vlan - https://phabricator.wikimedia.org/T184723#4290426 (10akosiaris) p:05Lowest>03Triage a:03ayounsi [10:35:10] 10Traffic, 10Operations, 10Page-Previews, 10RESTBase, and 2 others: Cached page previews not shown when refreshed - https://phabricator.wikimedia.org/T184534#4290582 (10Vgutierrez) [10:35:19] 10Traffic, 10Operations, 10Page-Previews, 10RESTBase, and 2 others: Cached page previews not shown when refreshed - https://phabricator.wikimedia.org/T184534#4290585 (10Vgutierrez) [10:41:03] moritzm: 0.11.3-1~bpo8+1+wikimedia1 --> 0.11.3-1~bpo8+1+wikimedia2 that's the way to go here? [10:41:11] bumping the wikemedia version number? [10:48:09] vgutierrez: I think so, just wondering why is wikimedia1 instead of the usual wmfX but I'm sure tehre is a reason ;) [10:49:13] yup.. in that aspect I'm just following otto naming [10:50:30] let's see how it behaves on cp1008 O:) [11:03:40] 10Traffic, 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, and 2 others: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#4290888 (10Vgutierrez) I've just tested a new build of librdkafka (0.11.3-1~bpo8+1+wikimedia2) on cp1008 that includes the new TLS configuration... [11:03:47] looking good <3 [11:32:54] elukey: I was about to add the new tls settings to the puppet varnishkafka define... but I found https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/437467/ [11:32:59] elukey: should I wait till that is merged? [12:21:18] hi and bye, I'm packing things up for travel soon [12:21:24] yell/text if something is urgent :) [12:22:06] (I saw ridiculous phab issue briefly, hopefully contained now) [12:24:59] bblack: safe travels! [12:41:25] vgutierrez: good question! I think that it will take a bit before we are able to merge, but yeah let's maybe wait for the next week, I hope to be able to merge after it [12:49:22] elukey: it would mean merging to another branch anyways, right? [12:49:46] ema: what do you mean? [12:50:49] elukey: nevermind, I thought the CR above was about the varnishkafka patch (incompatible w/ varnish 5.1) [12:51:12] ah! [12:51:22] nono I am trying to merge the module into ops/puppet [12:51:34] together with nginx and other two [12:51:40] nice [12:51:49] but it breaks puppet-merge if done as it is :D [12:51:55] so some workarounds are needed [13:01:28] <_joe_> elukey: I have a proposal for you not to break things [13:01:48] <_joe_> 1 - you add the nginx module to environments/production/modules/nginx [13:02:02] <_joe_> 2 - you remove the submodule from puppet [13:02:11] <_joe_> 3 you move the module to the final destination [13:02:31] <_joe_> unless puppet merge doesn't do the right thing and remove the stale submodule [13:05:15] _joe_ I didn't get exactly how the environments work, do 1. and 2. need any follow up in puppet for the new module location? [13:05:21] or it will be picked up transparently? [13:05:36] <_joe_> puppet first looks inside the env [13:05:39] <_joe_> then in the main [13:05:51] interesting [13:05:57] <_joe_> but you know, you have that fancy little thing called "puppet compiler" to check it works :D [13:06:54] 10Traffic, 10AbuseFilter, 10Analytics-Kanban, 10Data-release, and 13 others: Alert instrumentation returning 500 errors - https://phabricator.wikimedia.org/T184721#4291838 (10Aklapper) a:03ema [13:07:24] well my patches are a no op now for the compiler but they break puppet-merges, so only with it I am not that confident :D [13:08:02] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: pybal's "can-depool" logic only takes downServers into account - https://phabricator.wikimedia.org/T184715#4291867 (10Aklapper) p:05Lowest>03High [13:08:07] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: Alert instrumentation returning 500 errors - https://phabricator.wikimedia.org/T184721#4291865 (10Aklapper) p:05Lowest>03High [13:23:04] 10netops, 10Operations: Rack/Setup new codfw QFX5100 10G switch - https://phabricator.wikimedia.org/T197147#4291942 (10ayounsi) [13:26:30] 10netops, 10Operations: Rack/Setup new codfw QFX5100 10G switch - https://phabricator.wikimedia.org/T197147#4291949 (10ayounsi) [13:29:13] 10netops, 10Operations: Rack/Setup new codfw QFX5100 10G switch - https://phabricator.wikimedia.org/T197147#4291955 (10ayounsi) [14:16:10] 10Traffic, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar): Upgrade cache_text to Varnish 5 - https://phabricator.wikimedia.org/T184448#4292761 (10Aklapper) [14:42:01] so maybe I am misreading [14:42:03] but https://grafana.wikimedia.org/dashboard/db/varnishkafka?panelId=14&fullscreen&orgId=1&from=now-7d&to=now&var-instance=eventlogging [14:42:21] on cp5012 one varnishkafka instance for eventlogging is getting a ton of events [14:42:42] I only saw the errors first and I tried to restart it [14:42:46] but then I saw the traffic [14:44:50] I don't see anything out of the ordinary here https://grafana.wikimedia.org/dashboard/db/varnish-traffic-instance-breakdown?orgId=1&var-datasource=eqsin%20prometheus%2Fops&var-cache_type=text&var-server=All&var-layer=frontend [14:45:47] we have a rogue varnishkafka instance :D [14:46:19] so after the restart it dropped [14:46:29] let's see how it goes [14:46:53] k [14:46:57] Andrew had to restart a couple of instances this week IIRC, there might be some tuning to do [14:47:00] thanks! [14:55:26] elukey: so.. should I help while you finish the varnishkafka repo migration? :P [14:55:32] s/help/wait/g [14:55:44] arg /o\ [14:56:40] vgutierrez: it is done :) [14:56:47] you are free to make your changes [14:57:13] oh lovely [14:59:58] elukey: so.. the new librdkafka1 it's working as expected on cp1008, next week we can deploy it everywhere [15:00:20] ack [15:00:26] maybe the week after prague? [15:00:34] even the 25th yeah [15:01:01] I'd rather have a beer with you than a two hours outage :D [15:01:33] +1 [15:11:53] <_joe_> you are expected to deploy nothing during next week [15:12:12] <_joe_> we asked everyone else to freeze deployments [15:12:20] <_joe_> we can't be the ones breaking that [15:14:09] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560#4293332 (10Papaul) @ayounsi @BBlack I am getting the network error message below during install on both lvs2009 and lvs2010. Please advice. Thanks.... [15:16:18] 10Traffic, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560#4293344 (10Papaul) log on install2002 for lvs2010 DHCPDISCOVER from 00:0a:f7:f0:02:40 via 10.192.48.2 Jun 15 15:06:56 install2002 dhcpd[18272]: DHCPOFFER on 10.192.49.7 t... [15:38:56] elukey: hmmm eventslogging and statsv varnishkafka are supposed to be using plain text comms? [15:40:31] yep those do not carry "sensitive" data so we didn't focus on the first TLS rollout. But, authorization/authentication is of course a plus, it would be good in the next quarter to do the same that we did for webrequest [15:41:22] ack :) [15:47:18] elukey: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/440544/ [15:49:49] looks good! [15:50:37] nice :D [17:57:53] 10Traffic, 10DBA, 10Operations, 10Patch-For-Review: dbtree broken (for some users?) - https://phabricator.wikimedia.org/T162976#4293777 (10jcrespo) [17:57:56] 10Traffic, 10DBA, 10Operations: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#4293775 (10jcrespo) 05Open>03stalled This is stalled because tendril cannot work with multiple db backends. We would need to setup a different backend to support it- w... [18:50:51] 10Traffic, 10DBA, 10Operations: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#4293838 (10Krinkle) [18:51:06] 10Traffic, 10DBA, 10Operations, 10Availability: dbtree: make wasat a working backend and become active-active - https://phabricator.wikimedia.org/T163141#3187493 (10Krinkle)