[03:25:05] 10Traffic, 6Services, 6operations, 5Patch-For-Review: Decom parsoidcache cluster - https://phabricator.wikimedia.org/T110472#1992753 (10BBlack) [03:25:09] 10Traffic, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver, 10Parsoid, and 4 others: Decom parsoid-lb.eqiad.wikimedia.org entrypoint - https://phabricator.wikimedia.org/T110474#1992751 (10BBlack) 5Open>3stalled Re: timing - per https://lists.wikimedia.org/pipermail/wikitech-l/2016-Janua... [09:27:43] <_joe_> ema: me and moritzm were looking at rebooting the wdqs100x servers, that are set as backends of cache-misc [09:28:04] <_joe_> I'm not sure how cache-misc checks the backend health [09:28:15] <_joe_> maybe you looked at it recently? [09:28:21] <_joe_> does it check health at all? [09:33:07] <_joe_> 'backend_match' => '^wdqs', [09:33:11] <_joe_> 'probe' => 'wdqs', [09:33:13] <_joe_> ok [10:09:44] _joe_: I haven't actually looked at cache-misc at all :) [10:15:38] <_joe_> ema: hic sunt leones [10:21:05] _joe_: yeah, I can imagine. For the time being I do not even have to look for trouble, it finds me by itself. So I'll postpone my safari in cache-misc land... [10:21:31] <_joe_> trouble always finds you :P [10:22:02] _joe_: I hope you've read this BTW https://www.usenix.org/system/files/1311_05-08_mickens.pdf [10:22:12] (speaking of lions and trouble) [10:22:54] <_joe_> ahaha yeah he's very funny [10:22:56] <_joe_> I believe (but cannot prove) that PHP developers [10:22:59] <_joe_> have souls. [10:35:05] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 2 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#1993413 (10Addshore) >>! In T124418#1985370, @ori wrote: > Distribution of (indirect) callers of `HTMLCacheUpd... [11:09:32] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 2 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#1993526 (10daniel) >>! In T124418#1993413, @Addshore wrote: > In the last 20 minutes we had roughly 7000 edits... [13:38:19] 10Traffic, 6operations, 5Patch-For-Review: Create separate packages for required vmods - https://phabricator.wikimedia.org/T124281#1993797 (10ema) The initial WMF packaging of [[https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/varnish/libvmod-netmapper | libvmod-netmapper]] and [[https://... [14:00:04] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 2 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#1993865 (10BBlack) Well then apparently the 10/s edits to all projects number I found before is complete bunk... [14:02:34] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 2 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#1993869 (10BBlack) So, current thinking is that at least one of (maybe two of?) the bumps are from moving what... [15:36:46] 7Varnish, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Wikimedia-Fundraising, and 3 others: Replace or redo Special:RecordImpression - https://phabricator.wikimedia.org/T45250#1994077 (10Nemo_bis) I already see two incompatible interpretations of "redo" in the comments above, so it appear... [16:19:38] 10Traffic, 6operations, 5Patch-For-Review: Create separate packages for required vmods - https://phabricator.wikimedia.org/T124281#1994184 (10ema) Varnish 4 has [[https://github.com/lkarsten/libvmod-ipcast/blob/master/README.rst | std.ip() as a built-in function]] . We probably can skip porting libvmod-ipcas... [16:23:50] 10Traffic, 6operations, 5Patch-For-Review: Create separate packages for required vmods - https://phabricator.wikimedia.org/T124281#1994206 (10BBlack) @ema +1 [16:36:12] 10Traffic, 10MediaWiki-Cache, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner, and 2 others: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan - https://phabricator.wikimedia.org/T124418#1994252 (10BBlack) heh so: T113192 -> https://gerrit.wikimedia.org/r/#/c/258365/5 is probably the Jan 20 bump. [16:36:24] bblack! [16:36:27] we good to go [16:36:33] thanks! [16:36:56] updating our ticket... [16:40:17] 10Traffic, 6operations, 5codfw-rollout, 3codfw-rollout-Jan-Mar-2016: Ability to switch Traffic infrastructure Tier-1 to codfw manually - https://phabricator.wikimedia.org/T125510#1994278 (10Krinkle) [16:44:10] 10Traffic, 6Zero, 6operations, 5Patch-For-Review: Merge mobile cache into text cache - https://phabricator.wikimedia.org/T109286#1994305 (10BBlack) [16:55:50] 10Traffic, 6operations, 7Pybal: pybal etcd coroutine crashed - https://phabricator.wikimedia.org/T125397#1994342 (10BBlack) Just triggered another one directly and immediately. The sequence in this case is: 1. Merged https://gerrit.wikimedia.org/r/#/c/267230/ (removes 1/9 servers from dc=eqiad,cluster=cach... [18:00:00] 10Wikimedia-DNS, 6operations, 5Patch-For-Review: adding new languages to DNS langs.tmpl doesn't work until zone template is edited as well - https://phabricator.wikimedia.org/T97051#1994679 (10BBlack) For now, one way to workaround this is: Run authdns-update as normal, which takes care of some important st... [19:19:16] 7HTTPS, 6Research-and-Data, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, and 4 others: Set an explicit "Origin When Cross-Origin" referer policy via the meta referrer tag - https://phabricator.wikimedia.org/T87276#1995053 (10Rdicerb) [20:38:48] 7HTTPS, 6Research-and-Data, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, and 4 others: Set an explicit "Origin When Cross-Origin" referer policy via the meta referrer tag - https://phabricator.wikimedia.org/T87276#1995404 (10Rdicerb) This is just about ready, so sending it over to Tech/News.