[10:45:17] 10Domains, 10Traffic, 10DNS, 10Operations: Could not reach wikipedia from domain wikipedia.fi - https://phabricator.wikimedia.org/T230470 (104shadoww) [11:05:21] 10Domains, 10Traffic, 10DNS, 10Operations: Could not reach wikipedia from domain wikipedia.fi - https://phabricator.wikimedia.org/T230470 (10Vgutierrez) a quick check shows: ` willikins:~ vgutierrez$ host -t ns wikipedia.fi Host wikipedia.fi not found: 2(SERVFAIL) ` And from the whois output: ` Nameserver... [11:12:05] 10Domains, 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: Could not reach wikipedia from domain wikipedia.fi - https://phabricator.wikimedia.org/T230470 (10Vgutierrez) p:05Triage→03Normal [11:34:03] 10Domains, 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: Could not reach wikipedia from domain wikipedia.fi - https://phabricator.wikimedia.org/T230470 (10Vgutierrez) So, after adding the zone file for wikipedia.fi and the proper redirect rules: ` $ curl http://wikipedia.fi -o /dev/null -v 2>&1|gre... [11:43:45] 10Domains, 10Traffic, 10DNS, 10Operations, 10Patch-For-Review: Could not reach wikipedia from domain wikipedia.fi - https://phabricator.wikimedia.org/T230470 (104shadoww) Ok. Thanks for lighting fast fix for this @Vgutierrez! [12:28:51] 10netops, 10Operations: Cleanup confed BGP peerings and policies - https://phabricator.wikimedia.org/T167841 (10faidon) That's an awesome idea, nice! We can't advertise just the /23 + /48 from eqord as these would be more-specifics to what eqiad itself advertises - and thus all of the eqiad traffic would flow... [13:12:47] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:13:57] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:18:47] 10Traffic, 10Operations, 10serviceops, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [13:28:40] ema hi, https://github.com/wikimedia/puppet/commit/b16466be426c0c1ed18628d2bd7a4474788b8402 seems to have broken the phabricator class in labs [13:31:44] paladox: hi, can you elaborate? [13:32:10] Puppet seems to be failing in labs due to me not having https://github.com/wikimedia/puppet/commit/b16466be426c0c1ed18628d2bd7a4474788b8402#diff-322cab2d41fcc32e65cd1c6106171d8e set [13:32:29] but when i do set them, it fails with: "Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, secret(): invalid secret ssl/phabricator.key at /etc/puppet/modules/sslcert/manifests/certificate.pp:76:26 at /etc/puppet/modules/profile/manifests/tlsproxy/envoy.pp:90 on node [13:32:29] phabricator.phabricator.eqiad.wmflabs" [13:33:23] mmh [13:33:34] I did add a dummy key to labs/private for phab: https://gerrit.wikimedia.org/r/#/c/labs/private/+/529311/ [13:33:56] paladox: what did you set in hiera? [13:34:11] i set: [13:34:12] "profile::tlsproxy::envoy::ensure": absent [13:34:12] "profile::tlsproxy::envoy::sni_support": 'no' [13:34:18] "profile::tlsproxy::envoy::global_cert_name": phabricator [13:34:39] profile::tlsproxy::envoy::services same as what you did [13:34:56] paladox: what if you don't set anything at all in hiera? [13:35:00] 10Traffic, 10Operations, 10Wikidata, 10serviceops, and 4 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10abian) [13:35:05] it fails a puppet run [13:35:09] ack [13:35:20] then please try setting exactly what's in prod [13:36:20] ok [13:42:50] ema got me further, but fails on: [13:42:51] Error: /usr/local/sbin/x509-bundle --skip-root --skip-first -c /etc/ssl/localcerts/phabricator.discovery.wmnet.crt -o /etc/ssl/localcerts/phabricator.discovery.wmnet.chain.crt returned 1 instead of one of [0] [13:42:51] Error: /Stage[main]/Profile::Tlsproxy::Envoy/Sslcert::Certificate[phabricator.discovery.wmnet]/Sslcert::Chainedcert[phabricator.discovery.wmnet]/Exec[x509-bundle phabricator.discovery.wmnet-chain]/returns: change from notrun to 0 failed: /usr/local/sbin/x509-bundle --skip-root --skip-first -c /etc/ssl/localcerts/phabricator.discovery.wmnet.crt -o /etc/ssl/localcerts/phabricator.discovery.wmnet.chain.crt returned 1 instead of one of [0] [13:43:05] Is there any way we can disable this class on labs? [13:46:34] paladox: what's the labs host in question? [13:46:55] phabricator.phabricator.eqiad.wmflabs [13:47:27] paladox: please give me access to it [13:47:30] ok [13:49:02] ema done [13:59:13] paladox: I can't ssh into it [13:59:18] oh [13:59:46] oh! i added you to the wrong project [13:59:58] done [14:00:00] ema ^ [14:03:40] ema, vgutierrez, bblack: do you have a min to take a look at :https://gerrit.wikimedia.org/r/c/operations/puppet/+/529053. Its trivial enough I promise [14:03:52] not right now TBH [14:05:51] ok no p [14:11:45] 10Traffic, 10Elasticsearch, 10Operations, 10Discovery-Search (Current work), 10Patch-For-Review: Icinga check defined from LVS configuration for cloudelastic are borked - https://phabricator.wikimedia.org/T229621 (10Mathew.onipe) This issue is solved for now and cloudelastic checks for all ports have bee... [15:10:50] 10Traffic, 10Operations, 10Wikidata, 10serviceops, and 4 others: [Task] move wikiba.se webhosting to wikimedia cluster - https://phabricator.wikimedia.org/T99531 (10BBlack) As noted in T155359 - WMDE has moved the hosting of this to some other platform, including the DNS hosting (and we never had the whois... [16:22:31] 10Traffic, 10Operations, 10Patch-For-Review: Roll out Anycast RecDNS to more servers - https://phabricator.wikimedia.org/T228190 (10BBlack) I'm not sure if it goes as a subtask here, or of T167841 and/or T227808 - but recording here so we don't forget, from an earlier IRC conversation: As things stand, if e... [19:10:35] 10netops, 10Operations: Instability of the Level3 link between cr2-eqiad and cr2-esams - https://phabricator.wikimedia.org/T228827 (10ayounsi) Circuit is down again, opened ticket 16915334. Account rep replied to the thread and put their client support manager in the loop as well. [20:00:38] 10Traffic, 10Operations, 10Performance-Team, 10TechCom-RFC, and 4 others: Serve Main Page of WMF wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10Ladsgroup) We talked about this with @Tgr in the hackathon and one easy way to bypass the issue of the redirect loop is to serve the ma... [20:32:07] XioNoX: any action needed here re: Amazon peering AS# change? https://groups.google.com/a/wikimedia.org/d/msg/ops-maintenance/p230BX-V5R4/H4qBw37fCQAJ [20:33:48] cdanis: They never replied to our peering requests, so no :) [20:33:52] lol [20:33:54] got it [20:34:11] thanks for the head's up, we do have some sessions with them in other DCS [20:39:33] 👍 [21:12:59] 10netops, 10Operations: ospf link-protection - https://phabricator.wikimedia.org/T167306 (10ayounsi) [21:13:02] 10netops, 10Operations: Cleanup confed BGP peerings and policies - https://phabricator.wikimedia.org/T167841 (10ayounsi) [21:13:11] 10netops, 10Operations: Cleanup confed BGP peerings and policies - https://phabricator.wikimedia.org/T167841 (10ayounsi) [21:23:29] 10netops, 10Operations: Cleanup confed BGP peerings and policies - https://phabricator.wikimedia.org/T167841 (10ayounsi) Sounds good, final version, including both AS 65002 and AS 65001 as optional to keep it generic. Tested the regex using `show route aspath-regex "^(65002|65001)? 64600.*"` Will push IPv6 fir...