[00:45:35] 10Traffic, 10Commons, 10Operations, 10Thumbor, 10media-storage: ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION - https://phabricator.wikimedia.org/T170605#3442366 (10Jeff_G) New symptom: https://upload.wikimedia.org/wikipedia/commons/thumb/2/23/Ortega%2C_Juan_de_%E2%80%93_Tratado_subtilissimo_de_aritm... [06:47:59] 10netops, 10Operations, 10Patch-For-Review: deploy diffscan2 - https://phabricator.wikimedia.org/T169624#3442654 (10MoritzMuehlenhoff) Two things I noticed (or we can move these to a new task): - For runs where no changes appeared, there should be no output/mail - Maybe we should setup a separate cron job t... [07:55:06] gehel: hey, wdqs1002 seems to be set as "under maintenance": [07:55:07] -rw-r--r-- 1 smalyshev wikidev 0 Jul 15 09:18 /var/lib/nginx/wdqs/maintenance [07:55:14] any idea why? [07:58:25] I've set it as inactive in etcd meanwhile so that the lvs stop obsessing about it [08:21:05] ema: IIRC he's still on vacation this weelk [08:22:09] moritzm: thanks! [09:03:53] 10Traffic, 10Commons, 10Operations, 10Thumbor, 10media-storage: ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION - https://phabricator.wikimedia.org/T170605#3442863 (10Aklapper) @Jeff_G: Please always also describe what you see instead of only posting links. Those two links work for me and I have no ide... [09:20:55] 10Traffic, 10netops, 10Operations: Recurring varnish-be fetch failures in codfw - https://phabricator.wikimedia.org/T170131#3442897 (10ayounsi) Another note, some of those spikes don't match with the OSPF flaps. [09:34:04] 10netops, 10Operations, 10Patch-For-Review: deploy diffscan2 - https://phabricator.wikimedia.org/T169624#3442952 (10ayounsi) 05Resolved>03Open [09:52:39] 10netops, 10Operations: pfw-codfw still logging to indium - https://phabricator.wikimedia.org/T170622#3443015 (10ayounsi) 05Open>03Resolved a:03ayounsi Done! Reopen if any issues. [13:55:18] ema: is https://gerrit.wikimedia.org/r/#/c/348066/ still relevant? [13:55:24] I think you released .9 or something? [13:57:08] paravoid: correct, 1.13.9 is out. The fixes in the patch you mentioned have not been merged yet, so eventually we'll want to make a new version including them (once we've confirmed that instrumentation still works with the right puppet change and all). I'l closing the CR for now! [14:39:42] 10netops, 10Operations, 10Patch-For-Review: deploy diffscan2 - https://phabricator.wikimedia.org/T169624#3443854 (10ayounsi) @MoritzMuehlenhoff For your first item, I tested the following locally and it seems to work fine: https://github.com/XioNoX/diffscan2/compare/master...XioNoX:quiet Per IRC conversatio... [15:31:09] 10netops, 10Operations, 10fundraising-tech-ops, 10ops-codfw: codfw: rack frack refresh equipment - https://phabricator.wikimedia.org/T169643#3444056 (10Papaul) a:05Papaul>03ayounsi @ayounsi The wiring is complete on all 4 devices I didn't plug the fiber from pfw3a and pfw3b to cr1 and cr2 yet. Let me... [15:35:40] 10Traffic, 10Operations, 10Pybal: pybal should automatically reconnect to etcd - https://phabricator.wikimedia.org/T169765#3444099 (10ema) So at first sight it looked like T169893 fixed this issue, but that's not the case. In particular, after conf1001 had been rebooted today I've noticed that both lvs1003 a... [16:38:24] 10Traffic, 10Operations, 10Pybal: Icinga check for pybal HTTP connections to etcd - https://phabricator.wikimedia.org/T170847#3444578 (10ema) [16:40:42] 10Traffic, 10Operations, 10Pybal, 10monitoring: Icinga check for pybal HTTP connections to etcd - https://phabricator.wikimedia.org/T170847#3444606 (10Volans) [16:43:17] 10Traffic, 10Operations, 10Phabricator, 10Release-Engineering-Team (Kanban): Verify that the codfw lvs is configured correctly for Phabricator - https://phabricator.wikimedia.org/T168699#3444618 (10mmodell) Still waiting on merge of https://gerrit.wikimedia.org/r/#/c/355869/ [17:34:37] ema: still here? [17:34:44] not urgent, just wondering :) [18:22:07] 10Traffic, 10Discovery, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3445111 (10Smalyshev) [18:22:34] 10Traffic, 10Discovery, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3445124 (10Smalyshev) p:05Triage>03High [18:30:52] paravoid: halfway :) [20:14:13] 10Traffic, 10Discovery, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3445856 (10Smalyshev) Some [[ https://pivot.wikimedia.org/#webrequest/table/2/EQUQLgxg9AqgKgYWAGgN7APYAdgC5gQAWAhgJYB2KwApgB5YBO1Az... [20:24:42] 10Traffic, 10Discovery, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3445111 (10debt) Looks like the [[ http://ipinfo.io/AS20633/141.2.0.0/16-141.2.108.0/23 | IP ]] is from a University in Germany: Jo... [20:37:46] 10Traffic, 10Discovery, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3445917 (10Smalyshev) Yup, no idea who that might be though. It produces 4% of all(!) queries on the service for the last week, and... [20:42:53] 10Traffic, 10Discovery, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3445111 (10Lydia_Pintscher) FWIW I don't know either who it could be. [20:48:41] 10Traffic, 10Discovery, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3445111 (10EBernhardson) I don't know if it would work as well here, but other services in the cluster user a cluster-wide semaphor... [20:58:01] 10Traffic, 10Discovery, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3446032 (10Esc3300) Maybe @Gymel would know who. [21:02:52] 10Traffic, 10Discovery, 10Operations, 10Wikidata, 10Wikidata-Query-Service: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3446101 (10Smalyshev) @EBernhardson we have per-IP limits, and they work fine in 99% of cases. In this case, however, the query is... [21:25:18] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 2 others: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3445111 (10GWicke) {T167906} might help with this problem, especially if you can reduce the allowed concurrency to a fairly small value. [21:25:32] 10Traffic, 10Discovery, 10Operations, 10Wikidata, and 2 others: Need ability to block specific query sources for WDQS - https://phabricator.wikimedia.org/T170860#3446287 (10faidon) [21:31:50] 10Traffic, 10Operations, 10Performance-Team, 10TemplateStyles, and 3 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#3446323 (10Jdforrester-WMF)