[03:09:06] :D [16:04:24] spamming here to avoid spamming -ops: [16:04:26] Sep 12 16:00:26 lvs1015 pybal[12097]: [sessionstore_8081 ProxyFetch] WARN: kubernetes1006.eqiad.wmnet (enabled/partially up/pooled): Fetch failed (https://localhost/healthz), 30.001 s [16:04:30] Sep 12 16:00:26 lvs1015 pybal[12097]: [sessionstore_8081 ProxyFetch] WARN: kubernetes1005.eqiad.wmnet (enabled/partially up/pooled): Fetch failed (https://localhost/healthz), 30.001 s [16:04:34] Sep 12 16:00:48 lvs1015 pybal[12097]: [sessionstore_8081 ProxyFetch] WARN: kubernetes1001.eqiad.wmnet (enabled/down/not pooled): Fetch failed (https://localhost/healthz), 30.001 s [16:04:38] Sep 12 16:00:48 lvs1015 pybal[12097]: [sessionstore_8081 ProxyFetch] WARN: kubernetes1003.eqiad.wmnet (enabled/down/not pooled): Fetch failed (https://localhost/healthz), 30.001 s [16:04:42] Sep 12 16:00:49 lvs1015 pybal[12097]: [sessionstore_8081 ProxyFetch] WARN: kubernetes1002.eqiad.wmnet (enabled/down/not pooled): Fetch failed (https://localhost/healthz), 30.001 s [16:04:46] Sep 12 16:00:49 lvs1015 pybal[12097]: [sessionstore_8081 ProxyFetch] WARN: kubernetes1004.eqiad.wmnet (enabled/down/pooled): Fetch failed (https://localhost/healthz), 30.001 s [16:04:50] ^ that was the last round of pybal->k8s failures for all [16:48:35] gehel: are you gone already? [16:49:35] andrewbogott: yep, sorry, already off, but back in ~5h [16:50:01] gehel: ok, easy question which you can ask on ticket — if your budget allows for 4 servers instead of 3, do you want 4? [16:50:09] s/ask/answer/ [16:50:36] (And, sorry for the ping, I wasn't clear on if your email meant you were out until Tuesday or just until the morning) [20:52:53] andrewbogott: I assume the task where you are expecting the answer is T232654 ? [20:52:54] T232654: eqiad: three clouvirt-wdqs servers for WDQS testing - https://phabricator.wikimedia.org/T232654 [20:54:45] Gehel, there's a different procurement ticket, I can find it in a bit [20:55:10] Oh, this one probably: T232663 [20:55:25] I was wondering why the first one looked so empty [20:58:00] ah, yep, you found it :) [20:59:00] for what it's worth (and as I understand it), budget not spent on hardware doesn't usually stay attached to a given project or team. So if you think you'll want four you should just go ahead and get four. [21:00:26] andrewbogott: understood, but I think we need 3. We might discover in the future that 4 is better (well, more is always better), but that's pure speculation [21:00:39] I'm pretty sure we can move forward a long way with 3 test servers [21:01:07] ok! [21:01:17] I'll assign that task back to rob then [21:01:19] so let's give those 6k back to someone who actually needs them [21:01:32] and thanks for moving this forward! [21:01:58] * gehel now has to hire 2 engineers to actually do something with those servers [21:11:22] gehel: you can do it! Hiring is never fun for me, but its awesome when its done [21:16:52] bd808: i actually enjoy it quite a lot ! [21:17:14] But it always take forever to find the right person [21:17:37] I like interviewing people, but I hate evaluating resumes and interview feedback [23:44:03] We now have netflow data from 1/3 of the Dallas routers, will push to all of them tomorrow