[00:00:37] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['lvs1014.eqiad.wmnet'] ` and were **ALL** successful. [00:25:59] 10Traffic, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install lvs101[3-6] - https://phabricator.wikimedia.org/T184293 (10BBlack) Note https://gerrit.wikimedia.org/r/c/operations/puppet/+/511118 - I had to switch the lvs1015 cross-row ports for rows A and B (enp4s0f1 and enp5s0f0) backwards a... [00:30:39] 10Traffic, 10Operations, 10Discovery-Search (Current work): nginx is failing to restart on cloudelastic100[1-2].wikimedia.org. Will also fail on cloudelastic100[3-4] when restart is attempted. - https://phabricator.wikimedia.org/T223734 (10Krenair) It sounds like you've got sslcert::ocsp::init without acme_c... [08:01:48] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10Ankit-Maity) p:05High→03Triage Widespread occurrence, VPT threads and lots of users affected. No point making a "Me to... [08:06:10] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10JJMC89) @Marostegui did a restart on cp1081 at 2019-05-19T05:09 which helped. Most recently I'm getting the issue from cp1... [08:06:15] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10DannyS712) I just came to report that this was happening again [08:07:56] 10Traffic, 10Operations, 10Chinese-Sites: Try to visit some pages on zhwikiversity but get a 503 error - https://phabricator.wikimedia.org/T223762 (10RazeSoldier) 05Open→03Invalid [08:08:11] 10Traffic, 10Operations: HTTP 503 when viewing some JavaScript page with action=raw&ctype=text/javascript - https://phabricator.wikimedia.org/T223763 (10RazeSoldier) [08:11:31] 10Traffic, 10Operations: HTTP 503 when viewing some JavaScript page with action=raw&ctype=text/javascript - https://phabricator.wikimedia.org/T223763 (10JJMC89) [08:11:38] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10JJMC89) [08:11:46] 10Traffic, 10Operations, 10Chinese-Sites: Try to visit some pages on zhwikiversity but get a 503 error - https://phabricator.wikimedia.org/T223762 (10JJMC89) [08:11:54] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10JJMC89) [08:12:03] 10Traffic, 10Operations: HTTP 503 when viewing some JavaScript page with action=raw&ctype=text/javascript - https://phabricator.wikimedia.org/T223763 (10RazeSoldier) 05duplicate→03Invalid > 05:09 marostegui: varnish-backend-restart on cp1081 - [[ https://wikitech.wikimedia.org/wiki/Server_Admin_Log#2019-05... [08:12:55] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10springrollconan) The problem is happening in Chinese Wikipedia. Mainly when saving pages, sometimes accessing pages. Some... [08:14:24] 10Traffic, 10Operations: HTTP 503 when viewing some JavaScript page with action=raw&ctype=text/javascript - https://phabricator.wikimedia.org/T223763 (10Ankit-Maity) [08:14:33] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10Ankit-Maity) [08:18:22] 10Traffic, 10Operations, 10Chinese-Sites: Try to visit some pages on zhwikiversity but get a 503 error - https://phabricator.wikimedia.org/T223762 (10Wang_Qiliang) [08:24:41] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10JJMC89) @jijiki did a restart on cp1087 at 2019-09-19T08:13 which should help for now. [08:43:44] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10GerardM) The problem is happening in Wikidata [10:02:26] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10ReaperDawn) The problem is happening in Indonesian Wikipedia [10:13:17] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations: https://sv.wikipedia.beta.wmflabs.org/ has invalid certificate - https://phabricator.wikimedia.org/T202564 (10Reedy) 05Open→03Resolved a:03Reedy >>! In T202564#5051923, @Krenair wrote: > @Zoranzoki21: they're just cherry-picks... specif... [12:09:15] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10jijiki) @ReaperDawn @GerardM are you still getting 503s? [12:28:35] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10CDanis) For posterity: https://grafana.wikimedia.org/d/000000352/varnish-failed-fetches?orgId=1&from=1558225397000&to=155... [12:34:33] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10ReaperDawn) @jijiki No, it is alright now in id.wikipedia. [12:38:56] 10Traffic, 10Operations, 10Wikimedia-General-or-Unknown, 10User-DannyS712, 10Wikimedia-Incident: 503 errors for several Wikipedia pages - https://phabricator.wikimedia.org/T222418 (10CDanis) 05Open→03Resolved a:03CDanis Thanks! We now believe this is resolved. [15:46:58] 10HTTPS, 10Traffic, 10Beta-Cluster-Infrastructure, 10Operations: https://sv.wikipedia.beta.wmflabs.org/ has invalid certificate - https://phabricator.wikimedia.org/T202564 (10Krenair) a:05Reedy→03Krenair yes