[10:01:07] * ema has a terrible doubt [10:01:20] it might be that pybal.log is empty because pybal doesn't have anything to say [10:02:37] for a long time we've had machines constantly depooled (eg: cp1052 because of network card issues), thus pybal kept on saying stuff (host is down but depooled, blabla) [10:03:13] now we don't have any depooled hosts in text/upload/maps, thus pybal doesn't log anything as there's nothing to log... [10:04:01] eheheh [10:04:35] <_joe_> that's very possible [10:04:46] <_joe_> try to shut down a server and you'll see! [10:05:12] I've tried depooling/repooling a frontend and indeed pybal logged the event properly *facepalm* [10:06:52] well ok, yet another bug brilliantly fixed! [10:14:53] 10Traffic, 06Operations, 10Pybal, 13Patch-For-Review: pybal stops logging - https://phabricator.wikimedia.org/T160405#3101310 (10ema) 05Open>03Resolved a:03ema In the past we've always had a bunch of hosts constantly depooled because of various issues (eg: T148891) thus we got used to seeing frequen... [10:30:22] <_joe_> ema: see? [14:19:59] 10netops, 06Operations, 10fundraising-tech-ops: set up firewall policies for barium, lutetium, db1025, and indium replacement servers - https://phabricator.wikimedia.org/T159336#3102151 (10Jgreen) p:05Normal>03Unbreak! Raising priority because this blocks deprecating Precise in fundraising. [14:21:29] paravoid: when you get a free window of time, wanna talk about DNS repo/templating/discovery/lint/etc? [14:21:52] 10netops, 06Operations, 10fundraising-tech-ops: set up firewall policies for barium, lutetium, db1025, and indium replacement servers - https://phabricator.wikimedia.org/T159336#3102171 (10Jgreen) [14:23:57] I want to, but busy atm :( [14:26:58] 10netops, 06Operations, 10fundraising-tech-ops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3102204 (10Jgreen) [14:28:42] 10netops, 06Operations, 10fundraising-tech-ops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3064590 (10Jgreen) [14:30:07] 10netops, 06Operations, 10fundraising-tech-ops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3102213 (10Jgreen) [14:30:51] 10netops, 06Operations, 10fundraising-tech-ops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3064590 (10Jgreen) [14:30:53] 10netops, 06Operations, 10fundraising-tech-ops: reassign wmf7010/frpm1001 to host "civi1001.frack.eqiad.wmnet" - https://phabricator.wikimedia.org/T159342#3102217 (10Jgreen) [14:37:47] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Operations, and 2 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3102252 (10Pcoombe) @AndyRussG That sounds fine to me. [14:58:44] paravoid: ok, I'm gonna make a POC commit so I know better about the little details in the meantime... [15:29:11] 10netops, 06Operations, 10fundraising-tech-ops: Cleanup layer2 firewall config from pfw-eqiad - https://phabricator.wikimedia.org/T111463#3102464 (10Jgreen) [16:00:27] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Operations, and 2 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3102614 (10DStrine) I'm cool with this too. thanks! [16:52:20] 10netops, 06Operations, 10fundraising-tech-ops: reassign wmf7010/frpm1001 to host "civi1001.frack.eqiad.wmnet" - https://phabricator.wikimedia.org/T159342#3102920 (10faidon) [16:52:30] 10netops, 06Operations, 10fundraising-tech-ops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3102917 (10faidon) 05Open>03Resolved a:03faidon That's done now :) [17:08:23] 10netops, 06Operations, 10fundraising-tech-ops: deploy firewall policies for (barium,lutetium,db1025,indium) replacements (civi1001,frdev1001,frdb1002,frlog1001) - https://phabricator.wikimedia.org/T159336#3102993 (10Jgreen) Great, thank you!!! [17:43:22] 10netops, 06Operations, 10fundraising-tech-ops, 13Patch-For-Review: reassign wmf7010/frpm1001 to host "civi1001.frack.eqiad.wmnet" - https://phabricator.wikimedia.org/T159342#3103222 (10Jgreen) p:05Normal>03High [18:41:01] 10Traffic, 10DNS, 06Operations, 10Parsoid, 13Patch-For-Review: Separate subdomain for parsoid visual diff test service on ruthenium - https://phabricator.wikimedia.org/T159995#3103496 (10Dzahn) p:05Triage>03Normal [18:41:08] 10Traffic, 10DNS, 06Operations, 10Parsoid, 13Patch-For-Review: Separate subdomain for parsoid visual diff test service on ruthenium - https://phabricator.wikimedia.org/T159995#3085873 (10Dzahn) a:03Dzahn [19:19:21] 10Traffic, 10DNS, 06Operations, 10Parsoid, 13Patch-For-Review: Separate subdomain for parsoid visual diff test service on ruthenium - https://phabricator.wikimedia.org/T159995#3103645 (10Dzahn) @ssastry on the DNS and Varnish side this should be done now. It should now just be up to Nginx config on ruth... [19:22:26] 10Traffic, 10DNS, 06Operations, 10Parsoid, 13Patch-For-Review: Separate subdomain for parsoid visual diff test service on ruthenium - https://phabricator.wikimedia.org/T159995#3103650 (10ssastry) >>! In T159995#3103645, @Dzahn wrote: > @ssastry on the DNS and Varnish side this should be done now. It sho... [19:22:42] 10Traffic, 10DNS, 06Operations, 10Parsoid, 13Patch-For-Review: Separate subdomain for parsoid visual diff test service on ruthenium - https://phabricator.wikimedia.org/T159995#3103653 (10Dzahn) a:05Dzahn>03ssastry All good for now? Could you confirm the new names work for you and take the next step o... [19:23:12] 10Traffic, 10DNS, 06Operations, 10Parsoid, 13Patch-For-Review: Separate subdomain for parsoid visual diff test service on ruthenium - https://phabricator.wikimedia.org/T159995#3103657 (10Dzahn) Ah, i sent my comment before i saw your new one. Ok, cool. [22:43:21] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 06Operations, and 3 others: Purge Varnish cache when a banner is saved - https://phabricator.wikimedia.org/T154954#3104307 (10DStrine)