[00:00:20] PROBLEM - SSH on cp5006 is CRITICAL: Server answer: https://wikitech.wikimedia.org/wiki/SSH/monitoring [00:15:14] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:24:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [00:30:38] PROBLEM - SSH on bast5001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [00:34:41] (03CR) 10Juan90264: Adding and use square wordmark for trwikiquote (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/704170 (https://phabricator.wikimedia.org/T286133) (owner: 10Juan90264) [00:40:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [00:45:35] (03PS22) 10Juan90264: Adding and use square wordmark for trwikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/704170 (https://phabricator.wikimedia.org/T286133) [00:46:41] (03CR) 10jerkins-bot: [V: 04-1] Adding and use square wordmark for trwikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/704170 (https://phabricator.wikimedia.org/T286133) (owner: 10Juan90264) [00:48:55] (03PS23) 10Juan90264: Adding and use wordmark in trwikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/704170 (https://phabricator.wikimedia.org/T286133) [00:50:04] (03CR) 10jerkins-bot: [V: 04-1] Adding and use wordmark in trwikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/704170 (https://phabricator.wikimedia.org/T286133) (owner: 10Juan90264) [01:00:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [01:01:02] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [01:03:54] (03CR) 10Juan90264: "Hello drobson, first sorry to include you in another referene change in wordmark, but this time I don't understand why jenkinsbot is votin" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/704170 (https://phabricator.wikimedia.org/T286133) (owner: 10Juan90264) [01:10:16] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [01:19:26] RECOVERY - SSH on cp5006 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [01:21:31] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [01:25:50] PROBLEM - SSH on cp5006 is CRITICAL: Server answer: https://wikitech.wikimedia.org/wiki/SSH/monitoring [01:26:31] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [01:29:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [01:31:38] RECOVERY - SSH on bast5001.mgmt is OK: SSH OK - OpenSSH_7.4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [02:01:31] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [02:07:53] RECOVERY - SSH on analytics1069.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [02:12:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [02:13:18] RECOVERY - SSH on cp5006 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [02:19:22] PROBLEM - SSH on cp5006 is CRITICAL: Server answer: https://wikitech.wikimedia.org/wiki/SSH/monitoring [02:22:16] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [02:52:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [02:59:48] RECOVERY - SSH on cp5006 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [03:06:06] PROBLEM - SSH on cp5006 is CRITICAL: Server answer: https://wikitech.wikimedia.org/wiki/SSH/monitoring [03:08:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: (2) Blazegraph instance wdqs2001:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [03:16:06] (03PS4) 10Juan90264: Add WN as an alias to project namespace in Polish Wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/725132 (https://phabricator.wikimedia.org/T291344) [03:16:34] (03CR) 10Juan90264: [C: 03+1] Add WN as an alias to project namespace in Polish Wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/725132 (https://phabricator.wikimedia.org/T291344) (owner: 10Juan90264) [03:19:14] PROBLEM - Varnish HTTP upload-frontend - port 80 on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Varnish [03:19:28] PROBLEM - Ensure traffic_exporter for the tls instance binds on port 9322 and responds to HTTP requests on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [03:20:00] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [03:20:28] PROBLEM - Varnish HTTP upload-frontend - port 3121 on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Varnish [03:20:28] PROBLEM - Varnish HTTP upload-frontend - port 3123 on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Varnish [03:20:32] PROBLEM - Varnish HTTP upload-frontend - port 3122 on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Varnish [03:20:33] PROBLEM - Varnish HTTP upload-frontend - port 3124 on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Varnish [03:21:02] PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [03:21:56] PROBLEM - Varnish HTTP upload-frontend - port 3126 on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Varnish [03:22:22] PROBLEM - Varnish HTTP upload-frontend - port 3120 on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Varnish [03:22:44] PROBLEM - Varnish HTTP upload-frontend - port 3125 on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Varnish [03:23:00] PROBLEM - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [03:25:08] PROBLEM - Varnish HTTP upload-frontend - port 3127 on cp5006 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Varnish [03:42:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [04:00:23] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp5006 is OK: OK - Certificate *.wikipedia.org will expire on Tue 16 Nov 2021 11:59:59 PM GMT +0000. https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [04:01:13] RECOVERY - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp5006 is OK: HTTP OK: HTTP/1.0 200 OK - 25936 bytes in 1.049 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [04:01:22] RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp5006 is OK: HTTP OK: HTTP/1.1 200 Ok - 35198 bytes in 0.778 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [04:47:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [04:52:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [05:09:31] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [05:11:16] PROBLEM - SSH on analytics1069.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [05:19:31] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [05:37:42] PROBLEM - Backup freshness on backup1001 is CRITICAL: Stale: 1 (gerrit1001), Fresh: 102 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [05:38:31] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [05:42:22] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [05:53:31] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [05:56:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [06:00:02] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp5009 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86397 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:04] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp5015 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86396 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:08] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1079 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86392 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:08] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1083 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86392 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:12] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp3062 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86388 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:16] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1077 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86384 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:18] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp5007 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86383 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:20] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1083 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86381 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:20] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp4030 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86380 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:20] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp4029 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86380 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:22] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp5008 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86378 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:28] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp3056 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86373 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:28] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp3060 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86373 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:28] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp3058 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86373 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:28] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp3054 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86373 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:28] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp3062 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86373 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:32] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp2031 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86369 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:32] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp2041 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86368 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:33] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp2029 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86368 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:40] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp2037 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86360 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:44] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1087 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86356 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:46] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1089 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86355 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:46] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1085 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86354 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:48] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp5008 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86352 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:50] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp2029 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86351 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:52] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp2039 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86347 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:54] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp2035 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86347 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:00:56] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp2027 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86344 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:00] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp5012 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86340 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:00] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp2033 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86340 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:00] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp2035 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86340 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:02] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp5011 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86338 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:02] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp5009 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86338 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:02] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp5010 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86338 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:02] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1077 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86338 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:04] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp4028 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86336 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:06] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp3052 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86334 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:08] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp3060 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86333 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:12] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp5015 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86328 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:12] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp5011 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86328 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:14] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp5007 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86327 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:14] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp5016 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86327 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:18] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp2033 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86323 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:18] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp2027 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86323 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:26] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1085 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86315 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:28] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1081 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86313 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:28] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1079 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86313 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:28] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp4030 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86312 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:28] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp4032 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86312 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:28] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp4031 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86312 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:28] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp4032 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86312 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:28] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp4027 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86312 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:29] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp4029 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86312 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:32] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1075 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86308 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:34] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1087 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86306 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:40] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1089 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86301 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:40] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp3058 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86300 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:40] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp1075 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86300 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:40] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp5010 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86300 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:42] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp3064 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86299 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:42] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp3050 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86299 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:46] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp5012 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86294 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:46] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp4031 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86294 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:46] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp3056 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86294 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:46] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp3050 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86294 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:48] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp3052 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86293 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:48] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp2031 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86293 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:54] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp3064 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86286 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:54] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp3054 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86286 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:58] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp5016 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86283 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:58] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp2037 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86282 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:01:58] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp2041 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86282 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:02:00] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp2039 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86280 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:02:00] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp1081 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86280 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:02:10] PROBLEM - ats-tls HTTPS wikiworkshop.org ECDSA on cp4027 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86270 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:02:10] PROBLEM - ats-tls HTTPS wikiworkshop.org RSA on cp4028 is CRITICAL: SSL CRITICAL - OCSP staple validity for wikiworkshop.org has 86270 seconds left https://wikitech.wikimedia.org/wiki/HTTPS [06:06:16] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [06:06:54] wonderful, update-ocsp-all.log is empty [06:12:20] RECOVERY - SSH on analytics1069.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [06:17:46] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [06:20:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [06:42:46] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20211003T0700) [07:08:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: (2) Blazegraph instance wdqs2001:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [07:15:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [07:20:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [07:25:16] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [07:30:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [07:39:48] RECOVERY - Backup freshness on backup1001 is OK: Fresh: 103 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring [07:50:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [07:55:31] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [08:00:32] PROBLEM - HTTPS on apt1001 is CRITICAL: SSL CRITICAL - OCSP staple validity for apt.wikimedia.org has 86368 seconds left https://wikitech.wikimedia.org/wiki/APT_repository [08:05:31] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [08:10:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [08:23:30] !log elukey@puppetmaster1001 conftool action : set/pooled=no; selector: name=cp5006.eqsin.wmnet [08:23:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:24:25] !log powercycle cp5006 (unresponsive to ssh, remote tty available but not able to login as root, no prometheus metrics in hours) [08:24:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:25:28] RECOVERY - Check systemd state on search-loader2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [08:25:31] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [08:25:38] RECOVERY - Number of messages locally queued by purged for processing on cp5006 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Purged%23Alerts https://grafana.wikimedia.org/dashboard/db/purged?var-datasource=eqsin+prometheus/ops&var-instance=cp5006 [08:25:50] PROBLEM - Host cp5006 is DOWN: PING CRITICAL - Packet loss = 100% [08:25:58] this is me --^ [08:26:46] RECOVERY - Host cp5006 is UP: PING OK - Packet loss = 0%, RTA = 236.15 ms [08:26:58] PROBLEM - Check systemd state on cp5006 is CRITICAL: CRITICAL - degraded: The following units failed: logrotate.service,prometheus-trafficserver-tls-exporter.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [08:27:00] RECOVERY - Varnish HTTP upload-frontend - port 80 on cp5006 is OK: HTTP OK: HTTP/1.1 200 OK - 465 bytes in 0.588 second response time https://wikitech.wikimedia.org/wiki/Varnish [08:27:18] PROBLEM - puppet last run on cp5006 is CRITICAL: CRITICAL: Puppet last ran 14 hours ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:27:26] elukey: it's been flapping ssh for hours [08:27:28] PROBLEM - purged service on cp5006 is CRITICAL: CRITICAL - Expecting active but unit purged is inactive https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [08:27:38] RECOVERY - SSH on cp5006 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:27:44] PROBLEM - Webrequests Varnishkafka log producer on cp5006 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [08:28:06] PROBLEM - Ensure trafficserver_exporter is running for instance tls on cp5006 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint https://127.0.0.1:443/_stats --port 9322 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [08:28:12] RECOVERY - Varnish HTTP upload-frontend - port 3121 on cp5006 is OK: HTTP OK: HTTP/1.1 200 OK - 470 bytes in 0.450 second response time https://wikitech.wikimedia.org/wiki/Varnish [08:28:13] RECOVERY - Varnish HTTP upload-frontend - port 3123 on cp5006 is OK: HTTP OK: HTTP/1.1 200 OK - 470 bytes in 0.470 second response time https://wikitech.wikimedia.org/wiki/Varnish [08:28:16] RECOVERY - Varnish HTTP upload-frontend - port 3124 on cp5006 is OK: HTTP OK: HTTP/1.1 200 OK - 470 bytes in 0.450 second response time https://wikitech.wikimedia.org/wiki/Varnish [08:28:16] RECOVERY - Varnish HTTP upload-frontend - port 3122 on cp5006 is OK: HTTP OK: HTTP/1.1 200 OK - 470 bytes in 0.472 second response time https://wikitech.wikimedia.org/wiki/Varnish [08:28:36] RECOVERY - Varnish HTTP upload-frontend - port 3127 on cp5006 is OK: HTTP OK: HTTP/1.1 200 OK - 470 bytes in 0.450 second response time https://wikitech.wikimedia.org/wiki/Varnish [08:28:46] RECOVERY - Ensure traffic_exporter for the tls instance binds on port 9322 and responds to HTTP requests on cp5006 is OK: HTTP OK: HTTP/1.0 200 OK - 23418 bytes in 0.754 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [08:29:06] RhinosF1: yep it was in a weird state [08:29:06] RECOVERY - Check systemd state on cp5006 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [08:29:28] RECOVERY - Varnish HTTP upload-frontend - port 3126 on cp5006 is OK: HTTP OK: HTTP/1.1 200 OK - 470 bytes in 0.452 second response time https://wikitech.wikimedia.org/wiki/Varnish [08:29:36] RECOVERY - purged service on cp5006 is OK: OK - purged is active https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [08:29:52] RECOVERY - Webrequests Varnishkafka log producer on cp5006 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [08:30:14] RECOVERY - Ensure trafficserver_exporter is running for instance tls on cp5006 is OK: PROCS OK: 1 process with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint https://127.0.0.1:443/_stats --port 9322 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [08:30:14] RECOVERY - Varnish HTTP upload-frontend - port 3120 on cp5006 is OK: HTTP OK: HTTP/1.1 200 OK - 470 bytes in 0.657 second response time https://wikitech.wikimedia.org/wiki/Varnish [08:30:32] RECOVERY - Varnish HTTP upload-frontend - port 3125 on cp5006 is OK: HTTP OK: HTTP/1.1 200 OK - 470 bytes in 0.486 second response time https://wikitech.wikimedia.org/wiki/Varnish [08:32:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [08:33:26] RECOVERY - puppet last run on cp5006 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:52:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [08:57:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [09:02:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [09:07:16] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [09:12:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [09:12:31] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [09:22:16] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [09:36:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [09:51:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [09:52:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [09:54:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [10:04:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [10:19:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [10:27:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [10:43:46] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [11:00:02] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir3002 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86399 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:00:14] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir4002 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86386 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:00:22] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir2001 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86378 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:00:58] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir5001 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86342 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:01:08] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir1002 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86332 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:01:18] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir1001 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86322 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:01:40] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir3001 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86301 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:01:40] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir5002 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86301 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:01:44] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir2002 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86296 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:02:04] PROBLEM - HTTPS non-canonical-redirect-4 on ncredir4001 is CRITICAL: SSL CRITICAL - OCSP staple validity for www.wikispecies.net has 86276 seconds left https://wikitech.wikimedia.org/wiki/Ncredir [11:08:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: (2) Blazegraph instance wdqs2001:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [11:11:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [11:17:54] PROBLEM - SSH on analytics1069.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [11:42:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [11:46:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [11:58:31] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:02:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:03:31] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:05:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:08:31] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:13:31] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:18:31] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:23:31] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:25:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:27:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:37:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs2008:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:47:01] (BlazegraphFreeAllocatorsDecreasingRapidly) resolved: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:51:21] PROBLEM - MariaDB Replica Lag: s7 #page on db1127 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 1180.78 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica [12:52:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: Blazegraph instance wdqs1009:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [12:52:48] here [12:53:25] same [12:53:31] i will ack the page [12:53:36] ty [12:54:08] <_joe_> kormat: replica seems to be broken [12:54:39] [5254042.655070] MCE: Killing mysqld:31372 due to hardware memory corruption fault at 7f3acf5a4042 [12:55:03] yeah we got bad ram [12:55:31] !log kormat@cumin1001 dbctl commit (dc=all): 'Depool db1127, bad ram', diff saved to https://phabricator.wikimedia.org/P17414 and previous config saved to /var/cache/conftool/dbconfig/20211003-125530-kormat.json [12:55:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:56:00] yay ecc [12:56:09] Previous RAM issues on that host in July: https://phabricator.wikimedia.org/T286763 [12:58:03] creating a new task. host is depooled. will disable notifications and ack alerts once i have a task to reference. [12:58:15] Thanks [12:58:18] <_joe_> perfect :) [12:58:40] thanks kormat [12:59:21] going back afk [12:59:29] 10ops-eqiad, 10DBA: Bad ram on db1127 - https://phabricator.wikimedia.org/T292366 (10Kormat) [13:00:08] (03PS1) 10Kormat: db1127: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/725450 (https://phabricator.wikimedia.org/T292366) [13:00:50] (03CR) 10Kormat: [C: 03+2] db1127: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/725450 (https://phabricator.wikimedia.org/T292366) (owner: 10Kormat) [13:07:43] (03PS1) 10Kormat: db1127: Use new notifications-disable format [puppet] - 10https://gerrit.wikimedia.org/r/725451 (https://phabricator.wikimedia.org/T292366) [13:09:24] (03CR) 10Kormat: [C: 03+2] db1127: Use new notifications-disable format [puppet] - 10https://gerrit.wikimedia.org/r/725451 (https://phabricator.wikimedia.org/T292366) (owner: 10Kormat) [13:13:58] ok, all done. [14:19:57] RECOVERY - SSH on analytics1069.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [14:45:03] <_joe_> !log restarting acmechief on acmechief1001 [14:45:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:47:09] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir5001 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 504771 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [14:48:37] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir5001 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 425483 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [14:49:27] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp4030 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486634 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:49:27] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp5009 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486633 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:50:01] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp5016 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486599 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:50:01] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp5009 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486599 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:50:35] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp4030 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486566 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:51:05] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp5016 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486535 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:52:01] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp3062 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486478 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:52:29] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir3001 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 425252 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [14:52:45] RECOVERY - HTTPS on apt1001 is OK: SSL OK - OCSP staple validity for apt.wikimedia.org has 493636 seconds left:Certificate apt.wikimedia.org valid until 2021-12-11 07:00:17 +0000 (expires in 68 days) https://wikitech.wikimedia.org/wiki/APT_repository [14:52:59] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir3001 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 504422 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [14:53:23] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp3062 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486397 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:53:37] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir1001 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 504383 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [14:54:13] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp5010 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486347 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:54:31] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp1085 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486330 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:54:35] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp5010 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486325 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:54:37] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp3056 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486324 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:55:15] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp3056 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486286 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:55:23] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp3050 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486277 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:55:35] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir1001 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 425067 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [14:55:37] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp1085 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486263 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:55:47] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp1083 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486253 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:55:59] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp3050 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486241 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:56:33] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp1083 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486208 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:57:25] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp3052 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486155 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:57:29] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir4002 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 504151 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [14:57:47] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir4002 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 424933 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [14:57:53] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp3052 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486127 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:57:57] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp3064 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486123 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:57:59] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp5011 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486122 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:58:27] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp5008 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486094 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:58:29] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir5002 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 504091 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [14:58:41] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp3064 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486079 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:58:51] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp5008 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486069 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:59:05] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp5011 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 486056 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [14:59:17] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir5002 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 424844 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:01:51] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp4028 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485890 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:01:57] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp2027 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485883 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:02:43] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp4028 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485837 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:02:59] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp2027 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485821 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:02:59] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp2035 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485821 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:02:59] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir2002 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 503821 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:03:07] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp5015 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485813 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:03:19] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp2035 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485801 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:03:29] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir2002 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 424592 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:03:31] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir3002 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 503789 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:03:31] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp1077 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485789 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:03:47] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp1075 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485773 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:03:49] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir3002 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 424571 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:03:59] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp5015 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485761 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:04:05] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp1075 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485755 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:04:07] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp1089 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485754 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:04:09] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp1077 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485751 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:04:15] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp2037 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485746 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:04:35] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp2037 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485725 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:04:51] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp5012 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485710 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:04:51] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp3058 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485709 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:04:57] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp5012 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485703 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:04:57] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp2039 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485702 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:05:09] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp1089 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485691 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:05:13] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp2039 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485687 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:05:15] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp3058 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485685 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:06:15] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir1002 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 503625 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:06:29] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp4031 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485611 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:06:43] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp4031 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485597 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:06:45] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp2031 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485595 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:07:21] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir1002 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 424359 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:07:39] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp2031 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485542 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:08:01] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp3060 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485520 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:08:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: (2) Blazegraph instance wdqs2001:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [15:08:39] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp3060 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485482 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:08:47] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp2029 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485474 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:08:53] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp2029 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485467 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:09:01] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir4001 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 424259 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:09:11] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp1087 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485449 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:09:29] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir4001 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 503431 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:09:45] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp1079 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485415 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:09:49] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp1079 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485411 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:09:55] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp1087 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485405 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:10:23] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp4027 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485377 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:10:23] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp3054 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485377 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:10:33] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp4027 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485367 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:11:45] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp3054 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485295 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:12:55] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp4029 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485226 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:13:31] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp5007 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485189 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:13:35] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp1081 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485186 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:13:43] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp1081 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485177 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:13:57] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp2041 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485164 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:14:23] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp5007 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485137 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:14:37] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp4032 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485124 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:14:37] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp4029 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485124 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:14:37] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp4032 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485124 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:14:49] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp2041 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485112 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:14:53] RECOVERY - HTTPS non-canonical-redirect-4 on ncredir2001 is OK: SSL OK - OCSP staple validity for www.wikispecies.net has 503107 seconds left:Certificate *.wikispecies.net valid until 2021-12-16 10:01:04 +0000 (expires in 73 days) https://wikitech.wikimedia.org/wiki/Ncredir [15:15:21] RECOVERY - ats-tls HTTPS wikiworkshop.org RSA on cp2033 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485079 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:16:01] RECOVERY - ats-tls HTTPS wikiworkshop.org ECDSA on cp2033 is OK: SSL OK - OCSP staple validity for wikiworkshop.org has 485040 seconds left:Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2021-12-05 17:00:17 +0000 (expires in 63 days) https://wikitech.wikimedia.org/wiki/HTTPS [15:16:17] RECOVERY - HTTPS non-canonical-redirect-3 on ncredir2001 is OK: SSL OK - OCSP staple validity for www.wikipedia.bg has 423823 seconds left:Certificate *.wikipedia.bg valid until 2021-12-25 12:01:30 +0000 (expires in 82 days) https://wikitech.wikimedia.org/wiki/Ncredir [16:18:23] PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The following units failed: netbox_report_accounting_run.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:44:03] RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [17:53:01] PROBLEM - SSH on ms-fe2006.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:08:16] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: (2) Blazegraph instance wdqs2001:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [19:24:57] PROBLEM - SSH on analytics1069.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:13:35] PROBLEM - Query Service HTTP Port on wdqs1004 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 298 bytes in 0.002 second response time https://wikitech.wikimedia.org/wiki/Wikidata_query_service [20:55:51] RECOVERY - SSH on ms-fe2006.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:11:16] (03CR) 10Bstorm: "I finally got around to this. The bullseye one works nicely. The node10 image is for Debian Stretch, which has no yarnpkg package, it seem" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/720788 (owner: 10Addshore) [21:16:21] (03PS1) 10Bstorm: openssl: update stretch container TLS libraries before using LE certs [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/725486 (https://phabricator.wikimedia.org/T291387) [21:19:42] (03PS1) 10Bstorm: Partially Revert "Add yarn to node images" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/725412 [21:22:15] (03PS2) 10Bstorm: Partially Revert "Add yarn to node images" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/725412 [21:23:33] (03CR) 10Bstorm: [C: 03+2] openssl: update stretch container TLS libraries before using LE certs [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/725486 (https://phabricator.wikimedia.org/T291387) (owner: 10Bstorm) [21:24:05] (03Merged) 10jenkins-bot: openssl: update stretch container TLS libraries before using LE certs [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/725486 (https://phabricator.wikimedia.org/T291387) (owner: 10Bstorm) [21:26:59] RECOVERY - SSH on analytics1069.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:37:38] (03CR) 10Bstorm: [C: 03+2] Partially Revert "Add yarn to node images" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/725412 (owner: 10Bstorm) [21:38:30] (03Merged) 10jenkins-bot: Partially Revert "Add yarn to node images" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/725412 (owner: 10Bstorm) [23:13:01] (BlazegraphFreeAllocatorsDecreasingRapidly) firing: (2) Blazegraph instance wdqs2001:9193 is burning free allocators at a very high rate - https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Free_allocators_decrease_rapidly - https://alerts.wikimedia.org [23:15:39] (03PS4) 10Juan90264: Wikiversity Logo Update for 2017 Logo Version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/725413 (https://phabricator.wikimedia.org/T292109) [23:28:13] PROBLEM - Elevated latency for icinga checks in codfw on alert1001 is CRITICAL: cluster=alerting instance=alert2001 job=icinga site=codfw https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/rsCfQfuZz/icinga [23:34:13] RECOVERY - Elevated latency for icinga checks in codfw on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/rsCfQfuZz/icinga