[08:38:10] 10Traffic, 10Operations, 10SRE-tools, 10Continuous-Integration-Config, and 5 others: Integrate automated DNS snippets into CI - https://phabricator.wikimedia.org/T243362 (10hashar) [08:54:05] 10Traffic, 10Operations, 10SRE-tools, 10Continuous-Integration-Config, and 5 others: Integrate automated DNS snippets into CI - https://phabricator.wikimedia.org/T243362 (10hashar) p:05Triage→03Medium [09:31:01] 10Traffic, 10Operations: Upgrade ncredir cluster to buster - https://phabricator.wikimedia.org/T243391 (10MoritzMuehlenhoff) 05Resolved→03Open I ran the OS upgrade tracking script and noticed that ncredir5002 is still on Buster, reopening [09:37:58] :O [09:38:05] moritzm: seriously? [09:38:10] my L8 is legendary [09:39:02] just think of the OS tracking script as L9 :-) [09:42:39] 10netops, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): CloudVPS: introduce filtering for neutron BGP addresses - https://phabricator.wikimedia.org/T246887 (10aborrero) p:05Triage→03High This filter might be blocking legit traffic. I believe we need to explicitly allow traffic with... [09:45:47] so I reimaged ncredir5002 and the DHCP config is the proper one, so I'm assuming puppet didn't run on time on the DHCP server [09:45:54] and the rest is history... and my L8 of course [09:55:13] we could simply make "server doesn't run buster" page in general :-) [10:05:40] "you just don't simply reimage a ncredir server to buster" [10:05:45] sigh [10:44:12] 10Traffic, 10Operations: Upgrade ncredir cluster to buster - https://phabricator.wikimedia.org/T243391 (10Vgutierrez) 05Open→03Resolved `vgutierrez@cumin1001:~$ sudo -i cumin 'A:ncredir' 'cat /etc/debian_version' 10 hosts will be targeted: ncredir[2001-2002].codfw.wmnet,ncredir[1001-1002].eqiad.wmnet,ncred... [10:53:02] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10aborrero) Reviewing this setup again, I just noticed this static route: ` route 185.15.57.0/29 next-hop 208.80.153.190; ` That CIDR is both the floati... [10:59:49] moritzm: solved :) [11:04:41] thx :-) [12:13:57] 10netops, 10Operations, 10cloud-services-team (Kanban): CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10aborrero) Wait, I think I may have a solution for this. [12:14:21] 10netops, 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): CloudVPS: introduce filtering for neutron BGP addresses - https://phabricator.wikimedia.org/T246887 (10aborrero) I think I may have a solution for this, stay tuned. [13:10:41] 10Traffic, 10Android-app-Bugs, 10Operations, 10Wikipedia-Android-App-Backlog, and 4 others: When fetching siteinfo from the MediaWiki API, set the `maxage` and `smaxage` parameters. - https://phabricator.wikimedia.org/T245033 (10JoeWalsh) a:03JoeWalsh [13:30:18] 10netops, 10Operations, 10Patch-For-Review: can aggregated netflow data include the router it was sampled from? - https://phabricator.wikimedia.org/T246186 (10elukey) If the cardinality of the three new dimensions are not too big we could definitely add them to Druid. One note - adding the new info to pmacc... [13:30:30] 10netops, 10Operations, 10Patch-For-Review, 10User-Elukey: can aggregated netflow data include the router it was sampled from? - https://phabricator.wikimedia.org/T246186 (10elukey) [13:51:15] 10Traffic, 10Continuous-Integration-Infrastructure, 10Operations: debian-glue-backports not enabling backports on buster - https://phabricator.wikimedia.org/T247316 (10ema) 05Open→03Resolved a:03ema >>! In T247316#5956625, @gerritbot wrote: > Change 578543 **merged** by Ema: > [operations/puppet@produc... [14:37:28] 10Traffic, 10Operations, 10observability: some Prometheis not scraping the full set of targets - https://phabricator.wikimedia.org/T246860 (10CDanis) 25h later, there's additional skew between 1003 and 1004: {P10686}