[00:26:21] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Joe) Another suggestion coming from twitter is https://play.google.com/store/apps/details?id=com.app.rcn, which anyways doesn't seem popular eno... [00:32:38] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Ladsgroup) I don't have much knowledge about India's internet infrastructure but from experience of Iran and blocking apps/websites. They show y... [00:59:42] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Preinheimer) Going to the TikTok website from India results in the regular TikTok page loading, with a banner from TikTok saying that the servic... [01:13:43] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Dzahn) https://newshimalaya.com/2021/02/09/%E2%9A%93-t273741-investigate-unusual-media-traffic-pattern-for-asternovi-belgii-flower-1mb-jpg-on-co... [01:36:00] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10varenc) >>! In T273741#6813823, @Preinheimer wrote: > Going to the TikTok website from India results in the regular TikTok page loading, with a... [03:17:10] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10tomglynch) Hi all, I've been doing a bit of research into possible apps that could be causing this and found two potential culprits that I am cu... [03:29:47] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10ssingh) Thank you everyone for the comments and suggestions. I just wanted to share that we have identified the app and will update this task to... [04:33:17] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10mmodell) >>! In T273741#6813839, @Dzahn wrote: > ^ wut? I tried to search for links to this image and found... this Phabricator ticket content... [04:34:07] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10mmodell) Also hello hacker news! [04:56:16] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10TheOv3rminD) >>! In T273741#6813995, @mmodell wrote: > Also, hello hacker news! https://news.ycombinator.com/item?id=26072025 Hello From us Hac... [07:28:11] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Zardula) Hello from hacker news [07:30:42] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Majavah) [07:31:08] I added a "please do not comment "hello from hacker news"" notice to that task ^ [07:43:52] 10netops, 10SRE: cr3-esams linecard diversity issue - https://phabricator.wikimedia.org/T262524 (10ayounsi) Disabling the interface on the asw2 side (via homer) `lang=diff [edit interfaces interface-range disabled] + member et-4/0/51; member ge-4/0/27 { ... } [edit interfaces interface-range disabled]... [07:45:01] 10netops, 10SRE: cr3-esams linecard diversity issue - https://phabricator.wikimedia.org/T262524 (10ayounsi) 05Open→03Resolved Remote hands did the recabling. Then pushed the cleanup via Homer as well. Everything is done. [08:03:12] 10netops, 10SRE: Upgrade Junos on asw2-esams - https://phabricator.wikimedia.org/T252631 (10ops-monitoring-bot) Icinga downtime set by ayounsi@cumin1001 for 1:30:00 32 host(s) and their services with reason: switch upgrade ` bast[3004-3005].wikimedia.org,cp[3050-3065].esams.wmnet,dns[3001-3002].wikimedia.org,g... [08:52:07] 10netops, 10SRE: Upgrade Junos on asw2-esams - https://phabricator.wikimedia.org/T252631 (10ayounsi) 05Open→03Resolved a:03ayounsi All done here. Note that we're hitting something similar to https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1363186 But it says "These log messages are ha... [09:00:48] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Shizhao) Rename to a new filename? [09:09:11] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10MoritzMuehlenhoff) [09:12:18] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10hashar) [09:20:29] 10Traffic, 10SRE: Create a generic network proformance profile - https://phabricator.wikimedia.org/T274230 (10jbond) [09:23:45] 10Traffic, 10SRE: Create a generic network proformance profile - https://phabricator.wikimedia.org/T274230 (10ayounsi) [09:23:57] 10Traffic, 10SRE: Create a generic network performance profile - https://phabricator.wikimedia.org/T274230 (10ayounsi) [09:31:15] 10Traffic, 10SRE: Create a generic network performance profile - https://phabricator.wikimedia.org/T274230 (10MoritzMuehlenhoff) Other classes in our Puppet tree which already apply some of the generic settings: * swift * profile::mediawiki::api * base::mysterious_sysctl * profile::mediawiki::common * profile... [09:31:27] 10Traffic, 10SRE, 10User-MoritzMuehlenhoff: Create a generic network performance profile - https://phabricator.wikimedia.org/T274230 (10MoritzMuehlenhoff) [10:48:08] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Vgutierrez) p:05Triage→03Medium [11:02:36] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Majavah) Phabricator doesn't also seem to be caching user profile pictures at all, which are also stored in the `phabricator_file` database. [11:36:37] 10Traffic, 10SRE, 10User-MoritzMuehlenhoff: Create a generic network performance profile - https://phabricator.wikimedia.org/T274230 (10jbond) I have created a starting point [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/662932 | profile ]] using mostly based on `cacheproxy::performance ` below ill... [16:01:33] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10AntiCompositeNumber) >>! In T273741#6814266, @Shizhao wrote: > Rename to a new filename? The Commons community generally avoids moving files, a... [16:01:56] 10Traffic, 10Commons, 10SRE: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Mvolz) [17:04:32] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10epriestley) > We need an easy way to tell logged-out traffic apart in our caching layer. Can your caching layer examine cookie values? If the `phsid` cookie... [17:17:44] 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10ssingh) Update: Thank you for the interest in this task! Like we shared yesterday, we have identified that the traffic is... [17:24:13] 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10fdans) @ssingh said it yesterday on chat but this is such stellar data detective work. Congrats on finding the culprit!! [17:25:57] 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Majavah) Is the effect that the block will have in the app known? [17:50:11] 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10calbon) Just to second what @fdans said, the data detective work was great and this was such a fun ticket to watch. [18:02:20] 10Traffic, 10Commons, 10SRE, 10Patch-For-Review: Investigate unusual media traffic pattern for AsterNovi-belgii-flower-1mb.jpg on Commons - https://phabricator.wikimedia.org/T273741 (10Joe) >>! In T273741#6815874, @Majavah wrote: > Is the effect that the block will have in the app known? No, hence we trie... [18:41:54] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Gilles) a:0320after4 [18:44:01] 10netops, 10Data-Persistence-Backup, 10SRE: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) **backup2002 -> backup1002** (please note this was while large backups were running in the backg... [18:46:49] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10mmodell) Indeed I think it's caching static files correctly. [18:47:45] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10epriestley) > Eventually Phabricator ran out of local sockets (30k limit): I'm not familiar with the particulars of your infrastructure, but if this was on th... [18:48:30] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Gilles) a:0520after4→03mmodell [18:52:18] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Majavah) Hmm. I'm looking at the Firefox developer console, and it looks like this: {F34098763} Note that styles and scripts are marked as "cached" instead of... [19:12:10] 10Traffic, 10Phabricator, 10SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10epriestley) I suspect you're seeing that because when you reload a page by issuing a "Reload" command in your browser, most (all?) modern browsers interpret th... [19:22:01] 10netops, 10SRE: tcp handshake failure between pfw3-eqiad and frlog1001:6514 - https://phabricator.wikimedia.org/T263833 (10ayounsi)