[10:14:55] 10Domains, 10Traffic, 10DNS, 10Operations: Donate wikiźródła.pl and wikisłownik.pl to the Foundation - https://phabricator.wikimedia.org/T240446 (10tomasz) 05Open→03Stalled [11:40:22] 10Traffic, 10Operations, 10Inuka-Team (Kanban), 10Patch-For-Review, 10Performance-Team (Radar): Code for InukaPageView instrumentation - https://phabricator.wikimedia.org/T238029 (10SBisson) A very special thanks to @phuedx for reviewing and merging the patch!! This task is now for @nshahquinn-wmf to te... [12:27:09] (19) lvs[2001-2006].codfw.wmnet,lvs[1013-1016].eqiad.wmnet,lvs[5001-5003].eqsin.wmnet,lvs[3005-3007].esams.wmnet,lvs[4005-4007].ulsfo.wmnet [12:27:09] ----- OUTPUT of 'apt-cache policy...|grep Installed:' ----- [12:27:09] Installed: 1.15.7 [12:27:14] _joe_: ^^ done [15:03:38] ema: FYI the makevm cookbook should support all Ganeti clusters now [15:04:10] so if you're creating new VMs in the PoPs let me know if it works fine of you encounter any issue [15:06:43] ack [15:31:55] 10Traffic, 10Operations: Setup netconsole on upload@esams hosts - https://phabricator.wikimedia.org/T242579 (10ema) 05Open→03Resolved a:03ema This is now done in prod. All upload@esams nodes are sending their kernel messages to a central host. See `journalctl -u netconsole` on ganeti3002. [15:32:00] 10Traffic, 10Operations: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10ema) [16:07:46] 10Traffic, 10netops, 10Operations: esams ipv6 reachability degraded - https://phabricator.wikimedia.org/T243127 (10CDanis) 05Open→03Resolved a:03CDanis This has recovered, but at its worst point we were down to just barely above 93% of probes with ipv6 reachability to esams. [16:08:34] 10Traffic, 10DC-Ops, 10Operations, 10ops-esams: Upgrade BIOS and IDRAC firmware on esams caches - https://phabricator.wikimedia.org/T243167 (10wiki_willy) a:03RobH [16:13:36] 10Traffic, 10DC-Ops, 10Operations, 10ops-esams: Upgrade BIOS and IDRAC firmware on esams caches - https://phabricator.wikimedia.org/T243167 (10wiki_willy) @RobH - can you work with the traffic to get the bios upgraded on the cp hosts in esams? In T240177, @Papaul found a Dell bulletin that associated wit... [16:48:08] 10Traffic, 10Operations, 10TechCom-RFC, 10Core Platform Team Legacy (Designing), 10Services (designing): Make API usage limits easier to understand, implement, and more adaptive to varying request costs / concurrency limiting - https://phabricator.wikimedia.org/T167906 (10chasemp) 05Open→03Declined L... [16:53:44] 10Traffic, 10DC-Ops, 10Operations, 10ops-esams: Upgrade BIOS and IDRAC firmware on esams caches - https://phabricator.wikimedia.org/T243167 (10RobH) a:05RobH→03BBlack Please note that #traffic (and @bblack) previously asked me NOT to do this on these hosts, while they worked out why they are crashing.... [18:40:46] 10netops, 10DC-Ops, 10Operations, 10Patch-For-Review: Juniper network device audit - all sites - https://phabricator.wikimedia.org/T213843 (10ayounsi) If we do abstraction of all the power supplies (that the CR above is for), there are still inconsistencies, but the list is progressively shrinking. Some ar... [19:19:53] 10netops, 10Operations: BFD session alerts due to inconsistent status on cr3-knams - https://phabricator.wikimedia.org/T240659 (10ayounsi) From JTAC: ` Been checking this issue with one of my seniors, on MX204 cr3-knams can we set on the below command: set routing-options ppm no-delegate-processing And cofig... [19:30:41] 10netops, 10Operations: BFD session alerts due to inconsistent status on cr3-knams - https://phabricator.wikimedia.org/T240659 (10ayounsi) Note that the above probably reset the sessions, as they are now up. Leaving it configured until the issue happen again. [20:19:29] 10netops, 10Operations: mr1-esams RMA (2020 edition) - https://phabricator.wikimedia.org/T242097 (10ayounsi) Errors are still there... I asked for the next recommended steps. I can imagine it's either another RMA or to try a new optic: https://www.fs.com/products/13272.html [20:37:16] 10netops, 10Operations: mr1-esams RMA (2020 edition) - https://phabricator.wikimedia.org/T242097 (10RobH) [20:42:28] 10netops, 10Operations: mr1-esams RMA (2020 edition) - https://phabricator.wikimedia.org/T242097 (10RobH) Next steps: * @robh and @wiki_willy to order another optic (or two) via T243335 . * @robh to file remote hands request with iron mountain, for when the optics arrive ** remote hands include swapping exist... [20:47:43] 10netops, 10Operations: mr1-esams i2c syslog flood - https://phabricator.wikimedia.org/T242097 (10ayounsi) [22:15:23] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712: Pages whose title ends with semicolon (;) are intermittently inaccessible - https://phabricator.wikimedia.org/T238285 (10MusikAnimal) I ran into this when I was unable to block https://en.wikipedia.org/w/index.php?target=Printf%28%2... [22:18:25] 10Domains, 10Traffic, 10DNS, 10Operations: Donate wikiźródła.pl and wikisłownik.pl to the Foundation - https://phabricator.wikimedia.org/T240446 (10Dzahn) @tomasz Is @CRoslof looped into the conversation with Doneva?