[01:17:51] 07HTTPS, 10Traffic, 06Discovery, 06Operations, and 2 others: Consider switching to HTTPS for Wikidata query service links - https://phabricator.wikimedia.org/T153563#2884289 (10Ricordisamoa) Is it advisable to use statements like `strafter(str(?item), str(wd:))` to avoid hard-coding URI prefixes within WDQ... [01:32:41] 10Traffic, 06Operations: Letsencrypt all the prod things we can - planning - https://phabricator.wikimedia.org/T133717#2948104 (10RobH) [01:32:46] 10Traffic, 06Operations, 13Patch-For-Review: convert librenms.wikimedia.org from GS to LE cert (expires: 2017-02-11) - https://phabricator.wikimedia.org/T154919#2948103 (10RobH) [07:40:20] 10netops, 06Operations: cr2-esams<->cr2-eqiad link flaps - https://phabricator.wikimedia.org/T154577#2948560 (10faidon) 05Open>03Resolved a:03faidon I was just on a lengthy phone call with Level3. This seems to have been a combination of issues with a 100G card ("fixed" by a card reset) that was done ori... [07:41:55] 10netops, 06Operations: Packet loss from Voxel to text load balancers - https://phabricator.wikimedia.org/T153998#2948566 (10faidon) 05stalled>03declined Since this was a user on IRC I doubt we'll hear much soon. Declining for now, feel free to reopen if the issue persists and we hear back from this or ano... [11:25:40] is there a Herald rule that adds #Traffic when HTTPS is mentioned or something? [11:25:53] https://phabricator.wikimedia.org/T155359 is interesting :) [11:40:28] paravoid: apparently yeah https://phabricator.wikimedia.org/H131 [11:40:35] the task has been created with #HTTPS project tag [11:40:47] and herald H131 automagically add #traffic to it [11:41:38] and later when the title got edited, another rule triggered which added #operations because there is #traffic [11:42:58] copy pasted on task [11:43:01] 07HTTPS, 10Traffic, 06Operations, 10Wikidata: wikiba.se should use HTTPS - https://phabricator.wikimedia.org/T155359#2948949 (10hashar) Per H131 whenever a task has the tag #HTTPS associated to it, #Traffic is automatically added. Then on a next edition #operations is added because #Traffic is present. [19:27:32] 10netops, 06Operations, 10ops-eqiad: asw2-d-eqiad.mgmt.eqiad - JNX_ALARMS CRITICAL - 2 red alarms, - https://phabricator.wikimedia.org/T152182#2950138 (10Cmjohnson) @faidon The QFXs mgmt ports are up and have a link light, they also do have 1 sfp management port. [19:50:27] <_joe_> ema: you just repooled ulsfo? [19:51:21] _joe_: ulsfo wasn't depooled, yesterday I've routed ulsfo back to codfw rather than straigth to eqiad [19:51:31] codfw is still depooled in DNS though [19:52:02] <_joe_> sorry, codfw [19:52:11] <_joe_> ok cool [19:52:41] the idea is to wait till the backends refill and then repool codfw in DNS too [19:52:51] <_joe_> makes sense [19:52:55] possibly not during codfw rush hours :) [20:06:15] lol [20:11:03] 10Traffic, 06Operations, 10Pybal: Unhandled pybal error causing services to be depooled in etcd but not in lvs - https://phabricator.wikimedia.org/T134893#2281050 (10Volans) I encountered a similar issue today, this is the log on when it started: ``` Jan 12 13:19:09 lvs2003 pybal[23011]: [pybal] INFO: [api_... [20:11:06] ulsfo going through may not perfectly refill it ever, or not in reasonable time [20:11:26] because ulsfo clients (asia) will have different hot items than central-US traffic, etc [20:11:39] under normal conditions it's getting a different blend of requests [20:12:00] we have only the backend in ulsfo going to the backend on codfw right? [20:12:18] right [20:12:22] the frontends in codfw has still the stale data since it was depooled [20:12:27] given that it didn't get restarted [20:13:36] they did get restarted [20:15:40] ah ok, then we'll have a cold frontend and a partially warm backend :-P [20:22:01] yeah that's ok [20:22:18] frontends fill in the hot objects prety rapidly, and the misses are over the local network to the local backends [20:23:09] mostly the fe/be split within a DC is just about set-size vs traffic-handling [20:23:48] the FEs get a nice even split of traffic to handle the edge-side traffic levels well without coalescing a popular article all on one interface, etc... but because of that they're small [20:24:06] and the backends have a much larger effective set size because they're chashed on the URLs first [20:26:28] I like the term Hot Object Cache for what we call frontends, although I keep on forgetting it [22:13:41] 10Traffic, 06Operations, 06Operations-Software-Development, 10Pybal: Unhandled pybal error causing services to be depooled in etcd but not in lvs - https://phabricator.wikimedia.org/T134893#2950697 (10Volans)