[00:03:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [00:05:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:15:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-enable] [00:23:04] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:35:09] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [00:43:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:41:57] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:45:16] PROBLEM - misc3 Current Load on misc3 is CRITICAL: CRITICAL - load average: 8.54, 4.89, 2.32 [01:47:14] !log reboot misc3 - high load [01:47:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:48:29] PROBLEM - misc3 SSH on misc3 is CRITICAL: connect to address 185.52.1.71 and port 22: Connection refused [01:51:21] PROBLEM - misc3 citoid on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:39] PROBLEM - misc3 electron on misc3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:10] PROBLEM - misc3 Disk Space on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:24] PROBLEM - misc3 zotero on misc3 is CRITICAL: connect to address 185.52.1.71 and port 1969: Connection refused [01:52:37] PROBLEM - misc3 restbase on misc3 is CRITICAL: connect to address 185.52.1.71 and port 7231: Connection refused [01:53:06] RECOVERY - misc3 SSH on misc3 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u1 (protocol 2.0) [01:54:10] RECOVERY - misc3 Disk Space on misc3 is OK: DISK OK - free space: / 19124 MB (79% inode=89%); [01:58:13] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 2.92, 0.70, 0.23 [01:59:17] RECOVERY - misc3 citoid on misc3 is OK: TCP OK - 0.006 second response time on 185.52.1.71 port 6927 [01:59:32] RECOVERY - misc3 electron on misc3 is OK: TCP OK - 0.110 second response time on 185.52.1.71 port 3000 [02:00:44] RECOVERY - misc3 zotero on misc3 is OK: TCP OK - 6.058 second response time on 185.52.1.71 port 1969 [02:01:03] RECOVERY - misc3 restbase on misc3 is OK: TCP OK - 0.108 second response time on 185.52.1.71 port 7231 [02:03:05] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [02:07:24] !log apt-get upgrade - misc3 [02:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:14:20] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1438 MB (5% inode=94%); [02:14:51] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [02:15:15] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [02:22:40] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:23:12] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:06:30] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [03:06:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [03:07:18] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 7.37, 9.20, 4.73 [03:08:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [03:08:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:09:15] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 1.86, 6.45, 4.26 [03:24:26] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [03:32:13] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:43:57] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [03:51:48] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:12:43] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [04:14:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [04:20:07] anyone else having the "There seems to be a problem with your login session; this action has been canceled as a precaution against session hijacking" problem? [04:43:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [04:45:51] Hello bookguy! If you have any questions, feel free to ask and someone should answer soon. [04:48:17] hi, i can't login my wikis. keep getting the following message: "There seems to be a problem with your login session; this action has been cancelled as a precaution against session hijacking. Go back to the previous page, reload that page and then try again." [04:50:01] i worked on my wikis almost every day. no problem until now. any idea how it can be fixed? many thanks in advance. [04:50:19] my miraheze account is also bookguy, fyi. [04:51:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [04:58:43] It's happened to me too, it started recently [04:59:08] Just an hour ago, I couldn't save an article and logged out and it gave me that [04:59:12] I wonder... [04:59:13] !ops [04:59:15] aww [06:13:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [06:21:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:26:36] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2366 MB (9% inode=94%); [06:36:06] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 3 minutes ago with 7 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [06:51:52] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:56:05] JohnLewis, paladox, PuppyKun, Reception123, SPF|Cloud, Zppix: paging all sysadmins, Per my talk page and me just trying to edit. Sessions aren’t working on meta so people can’t act logged in [06:57:13] @Operations many reports on discord and in scrollback as well [07:03:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [07:11:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:27:39] Hello Guest4! If you have any questions, feel free to ask and someone should answer soon. [07:27:53] Hello! [07:28:28] I'm getting an error while trying to log into my wiki today, wondering if I'm the only one [07:29:04] Guest4: we are aware, trying to find out what happened [07:29:11] "There seems to be a problem with your login session; this action has been canceled as a precaution against session hacking. Please resubmit the form." [07:29:18] Ah ok [07:29:23] I was worried it was just me [07:29:46] Thanks for letting me know :) [07:29:48] Guest4: when did you first get the error [07:29:54] a few hours ago [07:30:29] But I hadn't tried to log in in probably a day [07:31:06] I could still browse the wiki, but when I tried to save something it gave me an error and told me to log in again. [07:31:12] Now this login error [07:31:36] Actually I also got an error while trying to log out. It said I couldn't log out. Sorry I don't remember the full error message. [07:31:55] Guest4: Thanks, sounds like it’s what we’ve noticed. It looks to have started at about 04:45 - any action on a logged in account fails [07:33:37] Do you use LDAP auth? [07:33:59] Guest4: it’s whatever mediawiki core uses [07:34:14] 04:20 is now the first noted report [07:34:58] ah, so the built in auth [07:35:09] So it is a PHP error [07:35:46] You can check out a thread like this, it is old but the same logic still applies when it comes to session cookies: [07:35:47] https://www.mediawiki.org/wiki/Topic:Pjby0sdeg3e60rfy [07:35:48] [ [RESOLVED] Login Problem on Project:Support desk ] - www.mediawiki.org [07:36:02] Let me know if I can help somehow [07:39:28] https://phabricator.miraheze.org/T4995 [07:39:29] [ ⚓ T4995 Unable to act while logged in ] - phabricator.miraheze.org [07:43:19] cool thanks [07:44:35] Guest4: we’ve found some related errors starting 04:08 in the logs, we’re looking into it now [07:46:01] !log restarted redis-server [07:46:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [07:46:39] SantaRhino: we're back! [07:47:35] Reception123: [07:47:38] Yey [07:54:20] Guest4: resolved [08:13:46] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [08:21:49] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:26:13] PuppyKun, paladox, Reception123, SPF|Cloud: https://phabricator.miraheze.org/T4995#95028 [08:26:14] [ ⚓ T4995 Unable to act while logged in or log in ] - phabricator.miraheze.org [08:26:27] !log redis-server restarted [08:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:27:00] Reception123: what would cause the memory to fill that much [08:28:35] no idea [08:30:37] Reception123: could the discord issue where the deletion isn't actually being noted be related? [08:32:56] Reception123: 57592e8a26532c52833cfe6f can you check exception.og [08:33:00] yeah, since Redis does jobqueue [08:33:17] Reception123: yeah [08:33:21] unfortunately I won't be available from now on so you'll have to see the rest with paladox [08:33:48] Reception123: ok [08:43:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [08:55:32] !log disable puppet on misc2 [08:56:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:59:13] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 7 minutes ago with 0 failures [09:12:06] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [09:31:08] SPF|Cloud: echo isn't marking notifications as read [09:31:32] SPF|Cloud: down again [09:31:40] Sorry! We could not process your edit due to a loss of session data. [09:31:40] You might have been logged out. Please verify that you're still logged in and try again. If it still does not work, try logging out and logging back in, and check that your browser allows cookies from this site. [09:32:35] PuppyKun, PuppyKun: ^ [09:32:48] paladox: ^ [09:36:20] SantaRhino: does it work again? [09:37:17] SPF|Cloud: yep [09:37:31] what's going on [09:38:19] !log install cron on mics2 to restart redis each 15 minutes [09:38:22] SPF|Cloud: also https://phabricator.miraheze.org/T4995#95069 assigned to you, will let you update [09:38:23] [ ⚓ T4995 Redis server keeps failing due to unable to allocate memory ] - phabricator.miraheze.org [09:38:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [09:43:13] SPF|Cloud: should wiki creations be safe to try? [10:03:46] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100],Exec[ufw-allow-tcp-from-any-to-any-port-53] [10:11:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [10:43:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [10:51:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [11:23:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 2 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[ssh],Package[exim4],Service[postfix],Exec[ufw-logging-low] [11:31:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [12:23:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members] [12:31:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [12:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je5kQ [12:35:09] [02miraheze/services] 07MirahezeSSLBot 039f443b1 - BOT: Updating services config for wikis [12:42:46] I’m around [12:43:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [12:44:07] paladox: https://phabricator.miraheze.org/T4995#95094 [12:44:09] [ ⚓ T4995 Redis server keeps failing due to unable to allocate memory ] - phabricator.miraheze.org [12:44:26] SPF|Cloud: patch fixed it with a cron job restarting it every 15 mins [12:44:31] paladox: ^ [12:44:44] but not one could do anything logged in for around 3 hours [12:50:49] I doint know why it needs an incident report I already know why [12:51:49] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [12:54:52] paladox: because no one could edit logged in for 3 hours and any outage over a few minutes should have a report per policy [12:56:30] Ok [12:57:33] paladox: also check the subtask as im confused what state mediawiki has put the pages into on that wiki as redis powers JQ and JQ is used for deletion [13:00:07] I doint think I understand? [13:01:07] paladox: have you read the subtask [13:01:45] Oh [13:01:45] Looks deleted to me? [13:02:39] paladox: not if you try and move something to that page or reupload [13:02:46] Mediawiki says it exists [13:03:02] Yes [13:03:03] It looks deleted to me? [13:03:20] Have you tried after the user recreated and re deleted the page? [13:03:32] paladox: after it was redeleted [13:04:00] Ok [13:05:04] paladox: have you actually tried to reupload/move to that page? [13:06:26] Nope [13:06:39] paladox: then please do [13:06:50] As that’s when mw moans the page exists [13:08:57] I’m currently mobile so there’s nothing I can do atm. I’m about to have lunch then I can have a look as I’ll be with my laptop. [13:10:33] paladox: ok [14:13:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown): Package[openssh-server],Service[ssh],Package[exim4],Service[postfix] [14:15:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je5tl [14:15:17] [02miraheze/puppet] 07paladox 035ece212 - Lower redis memory [14:17:13] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:18:53] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [14:20:50] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:21:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [14:25:00] SantaRhino the user can press "Ignore any warnings" to bypass the "file exists already" warning. [14:25:03] As i just did [14:25:05] so it works [14:25:55] paladox: didn't earlier [14:26:07] Did you press it? [14:26:20] paladox: ye [14:26:59] Well that's strange [14:27:03] any ways it works now [14:27:06] paladox: what did you upload? [14:27:13] a file i had [14:27:49] paladox: can you move https://bgo.miraheze.org/wiki/File:841.png over https://bgo.miraheze.org/wiki/File:SantasSleigh.png with the redirect suppressed [14:27:50] [ File:841.png - Board Game Online Wiki ] - bgo.miraheze.org [14:27:52] [ File:SantasSleigh.png - Board Game Online Wiki ] - bgo.miraheze.org [14:28:19] I'm not allowed to unless the user carn't and requested i do it. [14:28:50] oh [14:28:52] that's you [14:29:18] done [14:29:28] A non-identical file already exists at "mwstore://local-backend/local-public/1/13/SantasSleigh.png". [14:29:29] oh [14:29:42] guess i'll go ahead and delete that (should have gone to archives) [14:29:53] paladox: redis magic does that [14:30:03] guess what was down when it happened [14:30:14] root@mw1:/mnt/mediawiki-static/bgowiki# ls -lah 1/13/SantasSleigh.png [14:30:15] ls: cannot access '1/13/SantasSleigh.png': No such file or directory [14:32:44] paladox: huh? [14:33:06] per the error i sent "A non-identical file already exists at "mwstore://local-backend/local-public/1/13/SantasSleigh.png"." [14:33:25] apparently the file dosen't exist even though the error says a non identical file exists already [14:33:53] paladox: that's kind of the issue, it keeps saying something is there when it isn't [14:34:08] that's why I needed you [14:41:28] SantaRhino done [14:42:49] paladox: https://bgo.miraheze.org/wiki/File:SantasSleigh.png [14:42:52] [ File:SantasSleigh.png - Board Game Online Wiki ] - bgo.miraheze.org [14:42:57] yes? [14:43:08] Is saying file doesn’t exist [14:43:35] I'm not sure why it didn't move the file, mind re-uploding it please? [14:44:18] Will do in a minute [14:45:30] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Je5qt [14:45:31] [02miraheze/puppet] 07paladox 03cc65a89 - Revert "Lower redis memory" This reverts commit 5ece212226d762702382462ee3b4c348895a253c. [14:45:33] [02miraheze/puppet] 07paladox 03b913737 - Revert "redis: Change maxmemory_policy to volatile-lfu" This reverts commit 3a55244ab7b840d159e1201d07ca42237201a5be. [14:50:33] paladox: The upload is an exact duplicate of the current version of File:SantasSleigh.png. !!!!!!!!!! [14:50:45] did you click ignore? [14:51:12] paladox: 9000 times [14:51:18] ????? [14:51:42] root@mw1:/mnt/mediawiki-static/bgowiki# find ./ -name SantasSleigh.png [14:51:42] ./1/13/SantasSleigh.png [14:51:45] heh [14:51:57] SantaRhino try now [14:52:26] paladox: I did say suppress the redirect as well [14:52:45] paladox: nope [14:52:54] nope? [14:53:25] paladox: still failed [14:53:26] SantaRhino it worked for me? [14:53:57] paladox: not for me [14:54:40] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-9102],Exec[git_clone_phabricator-extensions],Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-9090] [14:54:48] SantaRhino it looks like it worked? [14:55:02] paladox: nope that's not what I uploaded [14:55:09] ssh mhttps://bgo.miraheze.org/wiki/File:SantasSleigh.png [14:55:11] [ File:SantasSleigh.png - Board Game Online Wiki ] - bgo.miraheze.org [14:55:11] https://bgo.miraheze.org/wiki/File:SantasSleigh.png [14:55:13] [ File:SantasSleigh.png - Board Game Online Wiki ] - bgo.miraheze.org [14:55:19] looks like a slaigh to me [14:55:23] *sleigh [14:55:42] https://usercontent.irccloud-cdn.com/file/ie9Mx4tB/Screenshot%202019-12-18%20at%2014.55.34.png [14:55:49] paladox: ^ very christmassy [14:56:06] clear your cache [14:56:32] i see https://imgur.com/a/ahfeh3S [14:56:34] [ Imgur: The magic of the Internet ] - imgur.com [14:56:35] paladox: ?action=purge it works [14:57:24] paladox: can you remove the redirect you were supposed to supress https://bgo.miraheze.org/w/index.php?title=File:841.png&redirect=no? [14:57:25] [ File:SantasSleigh.png - Board Game Online Wiki ] - bgo.miraheze.org [14:57:47] done [14:58:18] !log apt-get upgrade - misc4 [14:59:24] paladox: bot loves you [14:59:31] !log restart logbot on misc1 [14:59:33] heh [14:59:36] !log apt-get upgrade - misc4 [15:01:05] !log test [15:02:46] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:42] !log test [15:10:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je5qj [15:10:19] [02miraheze/puppet] 07paladox 03145c122 - logbot: Make wiki_connection a non tuple https is now the default and configurable through scheme config, see https://github.com/mwclient/mwclient/pull/213/files [15:10:20] [ Allow URL scheme to be configured in a more intuitive way by mtausig · Pull Request #213 · mwclient/mwclient · GitHub ] - github.com [15:28:24] paladox: https://phabricator.miraheze.org/T4995#95102 [15:28:25] [ ⚓ T4995 Redis server keeps failing due to unable to allocate memory ] - phabricator.miraheze.org [15:28:33] new report [15:32:27] Users will need to sign back in [15:32:29] i had to [15:32:38] through Special:UserLogin [15:33:39] !log test [15:37:35] I'm going to run resetUserTokens.php which will log everyone out. [15:38:16] paladox: why? [15:38:46] Because it seems a ton of users are having issues with there sessions [15:39:07] paladox: k [15:39:38] that's been everyone at some point since 4am [15:43:44] though just tried to reset logbots and it didn't fix the issue [15:44:11] paladox: hmm [15:44:25] Hello kuma! If you have any questions, feel free to ask and someone should answer soon. [15:44:27] Any logs for logbot [15:44:31] Hi kuma [15:44:36] Hello, [15:46:27] I am here about how I am haveing trouble with logging into one of merahize wikis and it won't let me due to 'session hijacking' but I don't know what's it talking about [15:47:56] Guest87095 hi, we are looking into it :) (we've had several reports about it) [15:48:36] Oh, so it's some sort of issue on the other side, I thought I did something wrong by accident [15:49:14] Nah, it's us. We store sessions in redis but appears we had some issues this morning. [15:51:25] Alright, thanks for chatting with me, it has brought relief [15:53:50] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [15:55:45] paladox: updated the task, can you make an announcement on discord [16:00:22] paladox: do you have a plan? [16:00:31] also maybe update topic [16:00:39] SantaRhino i'm busy [16:00:46] trying to resolve it :) [16:01:45] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:16:57] paladox: https://phabricator.miraheze.org/T4995#95130 suggestion? [16:16:58] !log test [16:16:59] [ ⚓ T4995 Redis server keeps failing due to unable to allocate memory ] - phabricator.miraheze.org [16:17:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:17:24] It's working now, spoke with ramnode. [16:17:31] Who appeared to have fixed the issue [16:17:51] paladox: ok [16:20:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je5Oz [16:20:09] [02miraheze/services] 07MirahezeSSLBot 0358c4a09 - BOT: Updating services config for wikis [16:21:36] paladox: thanks for fixing, don't forget to update phab [16:24:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:24:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:28:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:28:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:32:11] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:32:45] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [16:34:09] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24646 bytes in 0.004 second response time [16:34:42] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:38:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je53W [16:38:18] [02miraheze/puppet] 07paladox 0345dccb9 - redis: increase memory [16:53:55] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_dns],Exec[ops_ensure_members] [17:01:21] !log [14:58:18] <+paladox> !log apt-get upgrade - misc4 [17:01:26] !log [14:59:31] <+paladox> !log restart logbot on misc1 [17:01:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:01:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:01:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [17:23:40] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_33 [+0/-0/±3] 13https://git.io/Je5sM [17:23:42] [02miraheze/mediawiki] 07paladox 036bb4d5c - Update Editcount [17:23:43] [02miraheze/mediawiki] 07paladox 03d7f837a - Merge branch 'REL1_33' of github.com:miraheze/mediawiki into REL1_33 [17:25:09] paladox: 1.34 is tommorow [17:25:10] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/Je5sS [17:25:11] [02miraheze/mediawiki] 07paladox 0334d69f6 - Update Tweeki [17:31:25] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±3] 13https://git.io/Je5sx [17:31:27] [02miraheze/mediawiki] 07paladox 03b4132b7 - Update Editcount and Tweeki [17:31:48] [02miraheze/mediawiki] 07paladox pushed 035 commits to 03REL1_34 [+0/-0/±8] 13https://git.io/Je5sh [17:31:50] [02miraheze/mediawiki] 07DannyS712 03c9c7e2e - NewPagesPager: Fix namespace query conditions Bug: T240924 Change-Id: I28d276cae0518386cac3f9d571ba09e9eff6678b (cherry picked from commit b390ef6e5825e8906667d7a755d70b3478ce47b7) [17:31:51] [02miraheze/mediawiki] 07thiemowmde 03fbc41be - media: Log and fail gracefully on invalid EXIF coordinates The $coord value is a value extracted from the EXIF section of an image file. We expect it to be a float, but there is no guarantee this is the case. It could, for example, be an empty string. I suggest this trivial fix. It does have the following effects: * Instead of logging a PHP notice when floor() [17:31:51] hits something that is not a number, I try to log something that's more useful for later, more in-depth debugging. Note this log call isn't necessarily meant to stay, but to find an even better fix for this issue. * I return the string as it is. If it's "foo", the user will see "foo" instead of "0° 0′ 0″ N", which wasn't helpful. Also note how wrong and misleading the PHPDoc block for this function was. Bug: T226751 [17:31:51] Change-Id: I1ca98728de4113ee1ae4362bd3e62b425d589388 (cherry picked from commit f6787ede2db29fcc2c1923e23eaa2e9bf86522a1) [17:31:53] [02miraheze/mediawiki] 07reedy 033d47388 - Update RELEASE-NOTES Change-Id: I10c0115d9f1591dd2e0ade8497b168247bf70c05 [17:31:54] [02miraheze/mediawiki] ... and 2 more commits. [17:33:25] SantaRhino ok [17:34:38] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±2] 13https://git.io/Je5GJ [17:34:39] [02miraheze/mediawiki] 07physikerwelt 03601cd67 - Mimic CURLOPT_POST in GuzzleHttpRequest The MWHttpRequest is implemented by the CurlHttpRequest class and also the GuzzleHttpRequest class. However, curl based rendering set the CURLOPT_POST which implies that the 'Content-Type' header defaults to 'application/x-www-form-urlencoded'. To homgonize the functionality this patch mimics the curl behaviour in Guzzle. [17:34:39] Bug: T232866 Change-Id: Id60a8de18e5f1e750a3bde23bd8b0deca4071165 (cherry picked from commit 5e3a0e73955d6324c5dd6e12fbe36d3ba203d9db) [17:39:22] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:43:20] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [17:55:17] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Je5G0 [17:55:19] [02miraheze/mediawiki] 07paladox 03b28bc7c - Update LinkTitles to 9e9d8dc32ed8605ee2c8eef9d299568699dadcdd [18:02:45] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Je5GX [18:02:46] [02miraheze/mediawiki] 07paladox 03bae4bdb - Update PortableInfobox to a01847f0bfd9b6be682ffc05d4d14dbe47c8a73a [18:12:55] good [18:13:14] * hispano76 greetings [18:37:06] Hi [18:37:31] paladox: what about that new wmf beta extension for infoboxes [18:38:54] You’ll need to make a request for that [18:39:12] who extension? [18:44:49] paladox: there is [18:45:01] https://phabricator.miraheze.org/T4965 [18:45:03] [ ⚓ T4965 Install (possibly as global/default) Extension:Capiunto ] - phabricator.miraheze.org [18:45:23] Given it’s been low on updates might be best to try it [18:45:53] Although it’d be nice to know why it isnt deployed to their prod yet [18:46:25] hispano76: wikimedia foundation [18:48:46] ah, capiunto [18:49:20] I enabled it yesterday, to see if I can learn about modules XD SantaRhino [18:51:50] hispano76: cool [20:08:41] PROBLEM - www.tradwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.tradwiki.org' expires in 15 day(s) (Fri 03 Jan 2020 08:05:53 PM GMT +0000). [20:08:46] PROBLEM - tradwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.tradwiki.org' expires in 15 day(s) (Fri 03 Jan 2020 08:05:53 PM GMT +0000). [20:08:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je5ce [20:08:59] [02miraheze/ssl] 07MirahezeSSLBot 031549208 - Bot: Update SSL cert for www.tradwiki.org [20:12:39] RECOVERY - www.tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.tradwiki.org' will expire on Tue 17 Mar 2020 07:08:49 PM GMT +0000. [20:12:44] RECOVERY - tradwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.tradwiki.org' will expire on Tue 17 Mar 2020 07:08:49 PM GMT +0000. [20:22:55] SantaRhino: its time to update your email sig just saying [20:27:19] Zppix: on? [20:27:34] SantaRhino: whatever email you used to email staff@ :P [20:28:47] Zppix: fixed [20:28:52] :P [20:29:41] Zppix: forgot i had that :) [21:06:53] i figured [21:16:45] Zppix: i’ll do ZppixBot stuff at some point over the holidays, i did nothing. Today at all apart from rest. [21:21:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je5Cr [21:21:49] [02miraheze/puppet] 07paladox 037032917 - Revert "redis: increase memory" This reverts commit 45dccb9fcc3a2917718e4a0df9dc55856f978d3f. [21:21:57] paladox: see discord [21:23:57] ok [21:47:12] !log restart redis-server [21:47:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:48:51] Hey JohnLewis [21:48:57] hey [21:49:11] Does that mean the cron job set by SPF|Cloud isnt working [21:49:13] JohnLewis: [21:49:23] wait, that cronjob is still on? [21:49:48] JohnLewis: i’ve not got a clue [21:49:59] it's off, but anyway no - a crontab to restart a service never works. [21:50:13] No one logged they stopped it JohnLewis and why [21:50:33] because that's awful practise and usually contributes more to a problem than it solves [21:50:45] I can understand why he did it, but I don't advocate it at all :) [21:51:07] JohnLewis: yeah, it was the best until someone with time and knowledge could fix the issue [21:51:09] I'm trying a more long term (well as long term as we need it) solution based on information paladox has given me [21:51:18] Good [21:51:27] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 10 minutes ago with 0 failures [21:51:50] Well it would be good if it was working after 17 hours bjt [21:51:51] But [21:52:40] JohnLewis: i’ll leave you [21:52:45] To update/reopen [21:52:47] https://phabricator.miraheze.org/T4995 [21:52:48] [ ⚓ T4995 Redis server keeps failing due to unable to allocate memory ] - phabricator.miraheze.org [21:53:03] * SantaRhino will probably sleep [22:08:30] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Je5W5 [22:08:32] [02miraheze/puppet] 07JohnFLewis 037a27657 - switch from a RDB to a AOF persistence state [22:08:52] I'll keep an eye on it but things seem stable now [22:09:13] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:49:40] JohnLewis: want me to keep status like it is? [22:50:12]