[12:28:35] 10Traffic, 10Operations, 10ops-ulsfo: cp4024 kernel errors - https://phabricator.wikimedia.org/T174891#3722819 (10BBlack) Ping @RobH - this hardware needs replacing. I guess diagnostics aren't perfect, and neither is the SEL, but clearly the node crashes out even during a fresh install. [13:10:31] 10Traffic, 10Operations: Migrate to nginx-light - https://phabricator.wikimedia.org/T164456#3722972 (10BBlack) Auditing production tlsproxy users for the switch to `light` in https://gerrit.wikimedia.org/r/#/c/386424/ shows no excess modules used on any of them, except for the expected lua+ndk on the cache hos... [14:33:34] 10Traffic, 10Operations, 10ops-ulsfo: cp4024 kernel errors - https://phabricator.wikimedia.org/T174891#3723313 (10RobH) Yep, I'll email our Dell reps today explaining our issues and asking for next steps. (Not sure if they'll want to try swapping the mainboard out or just the entire system.) [14:49:36] 10Traffic, 10Operations: Migrate to nginx-light - https://phabricator.wikimedia.org/T164456#3723386 (10BBlack) And the other side of this audit. Before we try to (carefully) switch to nginx-light, we need them all upgraded to the latest version so the dpkg-level replacement works sanely: ``` bblack@neodymium:... [15:29:31] Hello, have a question I was hoping someone might help me with. I’ve been helping analyze some Wikidata edits from contributors who weren’t logged in (just know their IP address). From our exploration, some IP addresses appear to be private addresses (e.g. “10.68.17.174”). Anyone known how this is happening/why? [15:31:28] hall1467: 10.68.x.x is generally from WMCS (formerly known as labs, where lots of bots and such run) [15:32:23] the traffic is internal to our infrastructure, but it doesn't really come from WMF production software, it comes from whatever software/bots users are running on WMCS [15:47:49] Okay thanks, that's insightful. If an edit is coming from 10.68.x.x, is it going to be a bot for sure? Or could some un-automated edits also be coming from there? [15:56:28] it should mostly be automated stuff, but I guess we can't rule out that user there is operating "manually"? [15:56:41] in any case, it's a user with a WMCS account and access, which is generally used for running bot-like software [15:57:16] The User-agent could be helpful here, but I don't know if the logs you're looking at have that. [15:57:22] hall1467: ^ [16:05:15] I see, interesting. How could I look at user-agent information? I'm not sure I'm familiar with using that. [16:18:10] 10Traffic, 10ORES, 10Operations, 10Scoring-platform-team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3723835 (10hoo) [17:42:16] bblack: ^ [18:21:01] hall1467: sorry, things are busy today :) [18:21:19] hall1467: User-Agent is an HTTP header sent with the request. what kind of logs are you looking at? [18:22:05] hall1467: hmmm I guess wiki-level logs like you said, just has IP address, hmmm [18:29:46] bblack: No worries! I'm looking at a wiki-level log for wikidata. [21:47:34] hall1467, UA information is private [21:48:07] additionally 10.68.17.174 doesn't appear to exist [21:54:55] Krenair: Thanks for letting me know it's private. Here are contributions from that IP: https://www.wikidata.org/w/index.php?limit=50&title=Special%3AContributions&contribs=user&target=10.68.17.174&namespace=&tagfilter=&start=&end= [21:55:50] yeah those entries are from almost 3 years ago, the instance that was at that IP no longer exists [21:56:17] and I wouldn't expect any UA data had been retained from it either