[00:05:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:07:35] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 8.855 second response time [00:10:01] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:36] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:19:47] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [00:20:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2607:5300:205:200::17f6/cpweb [00:20:25] [02mw-config] 07soratako commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/JvSQt [00:21:08] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.003 second response time [00:21:18] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvSQZ [00:24:49] [02miraheze/puppet] 07paladox 033f3c6e5 - mediawiki: Introduce support for apache as a backend [00:24:50] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [00:24:52] [02puppet] 07paladox opened pull request 03#1296: mediawiki: Introduce support for apache as a backend - 13https://git.io/JvSQn [00:25:00] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:25:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:25:18] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 8.210 second response time [00:26:08] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 18713 bytes in 0.523 second response time [00:36:33] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:800:1056::2/cpweb [00:37:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.77.107.210/cpweb, 2607:5300:205:200::17f6/cpweb [00:41:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:42:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:50:26] [02mw-config] 07soratako commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/JvSQ6 [00:58:33] [02mw-config] 07soratako commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/JvSQF [01:05:06] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:09:41] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.412 second response time [01:25:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvS7o [01:25:09] [02miraheze/services] 07MirahezeSSLBot 03e0960be - BOT: Updating services config for wikis [01:28:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JvS7i [01:28:05] [02miraheze/puppet] 07paladox 03ef4666d - Create apache.pp [01:28:07] [02puppet] 07paladox synchronize pull request 03#1296: mediawiki: Introduce support for apache as a backend - 13https://git.io/JvSQn [01:28:28] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:31:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvS7S [01:31:39] [02miraheze/puppet] 07paladox 036913205 - nginx: Experiment setting keepalive_timeout to a low value [01:33:12] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 6.237 second response time [01:50:21] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2642 MB (10% inode=93%); [01:55:11] Hello cnoTe75! If you have any questions, feel free to ask and someone should answer soon. [01:58:03] !log reset extensions/CreateWiki on test2 [01:58:05] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:58:23] Hey all, could someone fill me in on how I might go about turning a template field into a category for the created page? [01:58:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:03:01] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:14:36] PROBLEM - test2 Puppet on test2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 1 failures [02:14:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb [02:19:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:23:51] !log install links on test2 [02:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:59:19] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:59:35] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:03:49] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.362 second response time [03:03:55] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.706 second response time [04:15:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvSFN [04:15:09] [02miraheze/services] 07MirahezeSSLBot 0316955ca - BOT: Updating services config for wikis [05:28:35] Hello cnote! If you have any questions, feel free to ask and someone should answer soon. [05:31:25] Hello cnoTe100! If you have any questions, feel free to ask and someone should answer soon. [06:12:19] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 8 minutes ago with 5 failures. Failed resources (up to 3 shown) [06:12:30] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 56 failures. Last run 7 minutes ago with 56 failures. Failed resources (up to 3 shown): File[wiki.alirezaivaz.ir_private],File[wiki.glaucuslinux.org],File[wiki.glaucuslinux.org_private],File[wiki.senety.com] [06:14:11] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 28 failures. Last run 8 minutes ago with 28 failures. Failed resources (up to 3 shown): File[wiki.rmbrk.sk_private],File[vsfan.net],File[vsfan.net_private],File[infectowiki.com] [06:17:25] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [06:17:35] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:19:08] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:26:55] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3515 MB (14% inode=93%); [07:02:14] paladox: ping when you’re around [08:42:31] PROBLEM - bacula2 Bacula Static on bacula2 is UNKNOWN: NRPE: Unable to read output [08:46:49] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: no terminated jobs [09:46:57] RhinosF1 yes? [09:55:26] [02mw-config] 07paladox commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/Jv9q8 [10:12:03] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 51 failures. Last run 7 minutes ago with 51 failures. Failed resources (up to 3 shown) [10:16:42] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [12:26:18] [02mw-config] 07soratako commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/Jv9WC [12:35:47] [02mw-config] 07paladox commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/Jv9W9 [13:05:46] [02mw-config] 07soratako commented on pull request 03#2942: Change copyright license on wikilexicon.miraheze.org per T5318 - 13https://git.io/Jv98P [13:33:16] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-1/±0] 13https://git.io/Jv9BZ [13:33:18] [02miraheze/MatomoAnalytics] 07paladox 0307abd30 - Rename SpecialAnalytics -> SpecialMatomoAnalytics [13:33:19] [02MatomoAnalytics] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fN4LT [13:33:21] [02MatomoAnalytics] 07paladox opened pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:33:51] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv9BC [13:33:52] [02miraheze/MatomoAnalytics] 07paladox 03e31e0fa - Update extension.json [13:33:54] [02MatomoAnalytics] 07paladox synchronize pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:34:09] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv9B8 [13:34:10] [02miraheze/MatomoAnalytics] 07paladox 039f17744 - Update en.json [13:34:12] [02MatomoAnalytics] 07paladox synchronize pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:34:21] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv9BB [13:34:22] [02miraheze/MatomoAnalytics] 07paladox 0346477fc - Update de.json [13:34:24] [02MatomoAnalytics] 07paladox synchronize pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:34:33] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv9BR [13:34:35] [02miraheze/MatomoAnalytics] 07paladox 031465707 - Update es.json [13:34:36] [02MatomoAnalytics] 07paladox synchronize pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:34:45] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv9B0 [13:34:46] [02miraheze/MatomoAnalytics] 07paladox 03e77346f - Update fa.json [13:34:48] [02MatomoAnalytics] 07paladox synchronize pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:34:55] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv9BE [13:34:57] [02miraheze/MatomoAnalytics] 07paladox 030027a88 - Update fr.json [13:34:58] [02MatomoAnalytics] 07paladox synchronize pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:35:08] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv9Bz [13:35:09] [02miraheze/MatomoAnalytics] 07paladox 03b95d7e3 - Update pt-br.json [13:35:11] [02MatomoAnalytics] 07paladox synchronize pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:35:19] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv9Bg [13:35:20] [02miraheze/MatomoAnalytics] 07paladox 037aab5d0 - Update qqq.json [13:35:22] [02MatomoAnalytics] 07paladox synchronize pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:35:31] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv9B2 [13:35:33] [02miraheze/MatomoAnalytics] 07paladox 03b1adb49 - Update ro.json [13:35:34] [02MatomoAnalytics] 07paladox synchronize pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [13:38:02] miraheze/MatomoAnalytics/paladox-patch-2/b1adb49 - paladox The build passed. https://travis-ci.org/miraheze/MatomoAnalytics/builds/667257935 [13:39:00] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jv9BK [13:39:01] [02miraheze/MatomoAnalytics] 07paladox 035daab8c - Rename noanalytics to matomoanalytics-noanalytics See https://gerrit.wikimedia.org/r/#/c/translatewiki/+/583518/ [13:39:02] [ Gerrit Code Review ] - gerrit.wikimedia.org [13:39:03] [02MatomoAnalytics] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fN4LT [13:39:04] [02MatomoAnalytics] 07paladox opened pull request 03#21: Rename noanalytics to matomoanalytics-noanalytics - 13https://git.io/Jv9B6 [13:39:28] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jv9BX [13:39:30] [02miraheze/MatomoAnalytics] 07paladox 0349471bb - Update en.json [13:39:31] [02MatomoAnalytics] 07paladox synchronize pull request 03#21: Rename noanalytics to matomoanalytics-noanalytics - 13https://git.io/Jv9B6 [13:39:46] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jv9B1 [13:39:47] [02miraheze/MatomoAnalytics] 07paladox 03d228efb - Update fa.json [13:39:49] [02MatomoAnalytics] 07paladox synchronize pull request 03#21: Rename noanalytics to matomoanalytics-noanalytics - 13https://git.io/Jv9B6 [13:40:02] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jv9BS [13:40:03] [02miraheze/MatomoAnalytics] 07paladox 0312edb45 - Update pt-br.json [13:40:05] [02MatomoAnalytics] 07paladox synchronize pull request 03#21: Rename noanalytics to matomoanalytics-noanalytics - 13https://git.io/Jv9B6 [13:40:38] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jv9B9 [13:40:40] [02miraheze/MatomoAnalytics] 07paladox 03fc03edf - Update extension.json [13:40:41] [02MatomoAnalytics] 07paladox synchronize pull request 03#21: Rename noanalytics to matomoanalytics-noanalytics - 13https://git.io/Jv9B6 [13:43:10] miraheze/MatomoAnalytics/paladox-patch-3/fc03edf - paladox The build passed. https://travis-ci.org/miraheze/MatomoAnalytics/builds/667260584 [13:46:05] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:46:22] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:50:50] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 3.130 second response time [13:51:06] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.560 second response time [14:06:12] any sysadmin mind a look at something? [14:06:41] scratch that [14:07:13] hang on, something weird [14:07:43] i need one of ya [14:11:10] I found a wiki with a messed up user namespace and a potential copyright violation [14:34:20] 502 error on meta [14:37:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 2607:5300:205:200::17f6/cpweb [14:38:09] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:52] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 18656 bytes in 4.034 second response time [14:46:39] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:04:50] [02MatomoAnalytics] 07JohnFLewis closed pull request 03#21: Rename noanalytics to matomoanalytics-noanalytics - 13https://git.io/Jv9B6 [15:05:26] [02MatomoAnalytics] 07JohnFLewis closed pull request 03#20: Rename SpecialAnalytics -> SpecialMatomoAnalytics - 13https://git.io/Jv9Bn [15:10:30] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±11] 13https://git.io/Jv9ua [15:10:31] [02miraheze/MirahezeMagic] 07translatewiki 03039f4c6 - Localisation updates from https://translatewiki.net. [15:10:32] [ Main page - translatewiki.net ] - translatewiki.net. [15:10:33] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jv9uV [15:10:34] [02miraheze/MatomoAnalytics] 07translatewiki 03b7c9417 - Localisation updates from https://translatewiki.net. [15:10:35] [ Main page - translatewiki.net ] - translatewiki.net. [15:11:20] Uh, JohnLewis ^ [15:13:59] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jv9uP [15:14:00] [02miraheze/MatomoAnalytics] 07paladox 036a0cea2 - Revert "Localisation updates from https://translatewiki.net." This reverts commit b7c9417ccfde7fefec755091279fdb7a08dd1fd0. [15:16:12] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [15:21:31] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [15:23:06] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jv9zJ [15:23:07] [02miraheze/mw-config] 07Reception123 036b2b994 - add SpriteSheet per T4874 [15:23:11] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw3 [15:27:25] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 2 backends are down. mw1 mw2 [15:27:26] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw1 [15:27:39] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:27:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [15:28:27] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* [15:28:34] ^ RhinosF1 you can imagine how fun it is to run it on 8 servers :( [15:28:46] heh [15:28:52] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [15:29:04] RhinosF1: it better work now [15:29:12] Reception123: i'm re-writing overly duplicated code [15:29:22] RhinosF1: from where? [15:29:29] MirahezeLogbot: hello? [15:29:33] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* [15:29:47] Reception123: some a friend's built with me [15:29:52] do I really need to open another server and restart MHlogbot [15:30:13] I've added about 40 lines and somehow made the code about 45 lines shorter [15:30:22] wow [15:30:25] I am a logbot running on mon1. [15:30:25] Messages are logged to . [15:30:25] To log a message, type !log . [15:31:07] what? why did it say that? [15:31:08] Reception123 use salt [15:31:11] I didn't do anything [15:31:16] paladox: oh, I thought we didn't have salt anymore [15:31:22] its on puppet2 [15:31:29] paladox: ah ok, thanks [15:31:38] salt -E "mw.*" cmd.run "sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki" [15:31:40] paladox: also what's up with logbot? Why is it saying it's a logbot instead of logging stuff [15:31:50] um [15:32:06] Reception123: you said hello [15:32:16] 15:29:30 <+Reception123> MirahezeLogbot: hello? [15:32:21] RhinosF1: ohh, it actually answers that? [15:32:26] !log restart logbot on mon1 [15:32:31] i guess so [15:32:31] MirahezeLogbot: hello? [15:32:32] paladox: thanks, was just going to do that [15:32:43] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* [15:33:09] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:33:16] heh why has the logbot died on us [15:33:20] I'll manually log for now then [15:34:10] I am a logbot running on mon1. [15:34:10] Messages are logged to . [15:34:10] To log a message, type !log . [15:34:24] !log upgrade grafana on mon1 [15:35:08] paladox: why does it keep saying that but doesn't log? [15:35:27] PROBLEM - cp6 Stunnel Http for mw1 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:35:51] PROBLEM - cp7 Stunnel Http for mw1 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:36:06] Reception123 according to service logbot status, its hitting the api. [15:36:22] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:37:20] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:37:21] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:37:51] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:38:12] paladox: it recorded you logging the restart but not me.... [15:38:15] I guess it really hates me [15:38:17] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* [15:38:40] heh :P [15:38:49] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:04] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 2.761 second response time [15:39:21] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [15:39:28] paladox: could you try to log what I did? [15:39:32] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [15:39:34] !log test [15:39:36] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [15:39:43] and if it works I'm summoning MirahezeLogbot to a disciplinary hearing [15:39:45] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 18682 bytes in 2.794 second response time [15:39:46] for discrimination [15:40:02] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:40:31] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:34] paladox: it finally worked now for me, but no reply still [15:42:48] paladox: could it be something with mon1? [15:43:00] No, its due to the slowness [15:43:05] ah [15:43:27] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.639 second response time [15:43:29] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.098 second response time [15:43:57] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:44:16] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 8.163 second response time [15:44:57] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18682 bytes in 2.750 second response time [15:46:22] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.717 second response time [15:46:41] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 3.069 second response time [15:50:04] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.335 second response time [15:51:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:51:45] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:27] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:46] PROBLEM - cp8 Stunnel Http for mw2 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:49] RECOVERY - cp6 Stunnel Http for mw1 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15356 bytes in 2.479 second response time [15:52:54] RECOVERY - cp7 Stunnel Http for mw1 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15356 bytes in 4.750 second response time [15:53:14] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15356 bytes in 7.359 second response time [15:57:58] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 8.734 second response time [15:58:01] RECOVERY - cp8 Stunnel Http for mw2 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.363 second response time [15:59:49] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:02:28] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18713 bytes in 0.564 second response time [16:02:33] PROBLEM - test2 HTTPS on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 254 bytes in 0.140 second response time [16:04:03] * hispano76 greetings [16:04:10] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:04:56] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.003 second response time [16:05:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb [16:07:48] RECOVERY - test2 HTTPS on test2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 618 bytes in 0.184 second response time [16:08:59] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.371 second response time [16:10:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:21:54] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv92m [16:21:56] [02miraheze/mw-config] 07Reception123 03628247c - add sql to spritesheet [16:23:58] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv92C [16:23:59] [02miraheze/mw-config] 07Reception123 03bc2f0af - fix mistake regarding SQL [16:24:07] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [16:26:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:45:05] https://www.irccloud.com/pastebin/6G2I4eae/ [16:45:05] [ Snippet | IRCCloud ] - www.irccloud.com [16:45:12] ^ paladox does this mean it's not compatible with 1.34? [16:45:42] Possibly [16:46:14] paladox: should I maybe file a bug task then? [16:48:51] Yup [16:49:01] ok [16:50:39] PROBLEM - test2 Puppet on test2 is WARNING: WARNING: Puppet is currently disabled, message: Paladox, last run 8 minutes ago with 0 failures [16:51:56] paladox: heh what can I do if there's no working issue tracker :( [16:54:01] I'll try to contact in another way [16:54:38] Are they using github? [16:55:19] paladox: yes but issues is disabled [16:55:23] so I'll see if I can email the dev [16:55:36] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv9aS [16:55:38] [02miraheze/mw-config] 07Reception123 03fd2d85b - temporarily disable SpriteSheet [17:00:57] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:05:58] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 2.799 second response time [17:25:01] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [17:34:22] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:34:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb [17:43:44] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [17:43:55] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw4 [17:44:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:47:37] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [17:48:06] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [18:22:52] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:20] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18682 bytes in 5.926 second response time [18:47:43] [02miraheze/mediawiki] 07paladox pushed 0326 commits to 03REL1_34 [+0/-0/±59] 13https://git.io/Jv9oh [18:47:45] [02miraheze/mediawiki] 07reedy 03ff037a9 - Start 1.34.1 Change-Id: I0dddff5a6f50820551966db37061dbada8c19e5f [18:47:46] [02miraheze/mediawiki] 07brightbyte 0362b44d8 - User: better error message when getActorId fails. This changes User::getActorId() to include the user name and id when throwing an exception. This doesn't solve the problem reported in T211450, but should allow the the probelmatic user name to be identified. Bug: T211450 Change-Id: Ie83ce6ad6b5ef18ea44a52e204f580cd9c992148 (cherry picked from commit [18:47:47] db3e7f8b7e81eed410e6f82f939038768efd27ce) [18:47:48] [02miraheze/mediawiki] 07reedy 03476413e - Don't redefine MW_ENTRY_POINT in thumb.php if already defined Bug: T241340 Change-Id: I95914267bf22910391a54ec524ed11bc076f83e3 [18:47:49] [02miraheze/mediawiki] ... and 23 more commits. [18:51:25] Thanks paladox [19:58:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.161.32.127/cpweb [20:08:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:24:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb [20:29:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:30:03] Any sysadmin want to do https://phabricator.wikimedia.org/T240400 [20:30:03] [ ⚓ T240400 Write and send supplementary release announcement for extensions and skins with security patches (MediaWiki 1.31.7/1.33.3/1.34.1) ] - phabricator.wikimedia.org [20:30:20] We definately need to update Widgets [20:34:01] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jv9Xj [20:34:02] [02miraheze/mediawiki] 07paladox 037941784 - Update Widgets [20:34:26] your free to do pulls ya know :) [20:35:26] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:35:30] paladox: im mobile, can we check MobileFrontend [20:35:39] it's already updated. [20:36:10] Cool, we did global blocking and we don’t have WikibaseMediaInfo right paladox [20:36:16] yes [20:36:24] Great! [20:36:46] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:37:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.161.32.127/cpweb [20:40:13] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:41:45] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 7.471 second response time [20:41:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.161.32.127/cpweb [20:42:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:49:13] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18713 bytes in 0.530 second response time [20:56:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:59:03] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 2.670 second response time [21:41:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb [21:43:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb [21:46:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:47:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:58:31] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:03:05] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.957 second response time [22:11:32] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw1 [22:12:14] PROBLEM - cp8 Stunnel Http for mw2 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:12:41] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:03] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [22:15:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.161.32.127/cpweb [22:15:56] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [22:16:35] RECOVERY - cp8 Stunnel Http for mw2 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.247 second response time [22:17:09] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 1.224 second response time [22:19:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:19:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:00:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2001:41d0:800:1056::2/cpweb, 2607:5300:205:200::17f6/cpweb [23:01:00] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [23:01:39] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:02:08] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:02:59] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw6 [23:03:52] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.233 second response time [23:06:26] !log apt-get install fio on db4 [23:11:56] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:06] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 5.308 second response time [23:12:32] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:04] PROBLEM - cp7 Stunnel Http for mw1 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:13] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.313 second response time [23:14:35] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:16:24] Hello Advocatus_Diabol! If you have any questions, feel free to ask and someone should answer soon. [23:16:49] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.502 second response time [23:17:19] hey [23:17:41] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [23:18:16] so I'm trying to import from an XML dump, what do I have to type in the interwiki prefix field? [23:19:09] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:19:43] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:20:07] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:21:47] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.701 second response time [23:22:47] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:21] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18695 bytes in 0.629 second response time [23:24:16] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:24:30] RECOVERY - cp7 Stunnel Http for mw1 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15350 bytes in 0.040 second response time [23:24:35] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv9H5 [23:24:36] [02miraheze/dns] 07paladox 0342adfb7 - Add dbt1 to dns [23:25:01] The interwiki prefix can be whatever you want, or you can pick a value from Special:Interwiki [23:25:57] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [23:27:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2001:41d0:800:1056::2/cpweb [23:27:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:28:14] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [23:28:37] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.010 second response time [23:29:00] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv9Qv [23:29:01] [02miraheze/dns] 07paladox 036b6db75 - Remove dbt1 ipv6 address [23:29:56] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18695 bytes in 1.286 second response time [23:31:13] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw6 [23:32:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+1/-0/±0] 13https://git.io/Jv9Q3 [23:32:28] [02miraheze/puppet] 07paladox 03085ef39 - Install mariadb on dbt1 [23:32:29] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [23:32:29] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [23:32:31] [02puppet] 07paladox opened pull request 03#1297: Install mariadb on dbt1 - 13https://git.io/Jv9Qs [23:32:40] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 1.240 second response time [23:33:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv9Qc [23:33:30] [02miraheze/puppet] 07paladox 03a0e5473 - Update site.pp [23:33:31] [02puppet] 07paladox synchronize pull request 03#1297: Install mariadb on dbt1 - 13https://git.io/Jv9Qs [23:33:48] [02puppet] 07paladox closed pull request 03#1297: Install mariadb on dbt1 - 13https://git.io/Jv9Qs [23:33:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jv9QC [23:33:51] [02miraheze/puppet] 07paladox 03df22cac - Install mariadb on dbt1 (#1297) * Install mariadb on dbt1 * Update site.pp [23:33:53] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [23:33:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [23:35:21] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [23:38:16] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [23:39:22] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:40:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:41:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:43:43] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [23:44:50] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.019 second response time [23:45:07] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.820 second response time [23:45:48] PROBLEM - dbt1 MySQL on dbt1 is CRITICAL: Host '51.89.160.138' is not allowed to connect to this MariaDB server [23:47:05] Hello AdvocatusDiaboli! If you have any questions, feel free to ask and someone should answer soon. [23:48:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv9QQ [23:48:26] [02miraheze/puppet] 07paladox 03bd9f515 - db: Add dbt1 to file wall