[08:17:02] Amir1: how can parsercache be 4TB already :-/ [08:17:08] And that's not binlogs like last time [09:29:38] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1287365 <-- could I get a +1, please? remove 2 drained hosts from the rings for reimaging [09:30:00] Emperor: done [09:30:12] thanks :) [09:57:44] federico3: cezmunsta is trying to decom a host and he ran into: https://phabricator.wikimedia.org/P92521 (the host does exist in zarcillo) does this ring a bell? [09:57:58] looking [09:58:17] thanks [09:58:56] no, never seen depool fail like this, I'm investigating a bit [10:00:49] it's the fqdn vs hostname: it needs only db2152 - but I should add a check [10:01:08] another one of those :( [10:01:14] cezmunsta: ^ try again! [10:01:20] uh? [10:01:50] * cezmunsta retrying [10:01:56] federico3: another case of fqdn vs hostname in cookbooks [10:02:29] Proceeding now [10:02:54] +1 [10:02:54] we did work to move everything away from fqdn towards hostnames... [10:03:28] I always fail to remember if it is one or the other [10:04:59] we should be able to use hostnames by default [10:27:40] (actually if anything it's not accepting hostnames please tell me and I'll fix it) [10:30:08] federico3: wilco thanks [10:46:47] cezmunsta: tip: you can run run "dbctl config diff" on cumin while scripts are running to keep an eye on pending changes and if there's any leftover - usually most pool/depool and other cookbooks show the diff in stdout anyways [10:47:36] : ty [10:49:10] also diff/commit are a bit misleading terms: this is not a VCS and changes are not transactional. It's more "list pending changes" "regenerate and save configuration" and you can get changes from other people all mixed together [12:27:17] pc3 is now back repooled with new hw in codfw [12:46:44] marostegui: can i start the switchover of s8 in codw? (I wanted to use the switchover helper but some deps are not ready at the moment, I'll try to use for s4 or s1) [12:47:03] sure, go for it [13:54:19] you may see some lag on backup1- hosts [13:54:33] you can ignore it, I am doing some heavy queries there [13:54:48] I will downtime the hosts, but may happen for some time [16:30:02] marostegui: I'm out today (some german holiday idk). The PC thing is because of deployment of parsoid for reads in large wikis, so we now have basically two entries for every page until they expire and rotate out (obviously a lot of binlog too) [16:30:47] Amir1: thanks, we can chat on Monday!