[00:09:57] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 50945MiB (11% inode=99%); [00:12:13] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49264MiB (10% inode=99%); [00:29:30] [02ssl] 07asko1 opened pull request #857: Redirects for citywikiwiki and bunkshelfwiki (07miraheze:03main...07asko1:03redirects1) 13https://github.com/miraheze/ssl/pull/857 [00:30:28] [02ssl] 07Universal-Omega merged 07asko1's pull request #857: Redirects for citywikiwiki and bunkshelfwiki (07miraheze:03main...07asko1:03redirects1) 13https://github.com/miraheze/ssl/pull/857 [00:30:28] [02ssl] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/0f24caa20125d3849d78d22b6bce35ab2b32933f [00:30:29] 02ssl/03main 07Asko 030f24caa Redirects for citywikiwiki and bunkshelfwiki (#857)… [00:34:53] [02ssl] 07coderabbitai[bot] commented on pull request #857: Two new redirect entries were added to the `redirects.yaml` configuration file. These entries define HTTPS redirects for `citywiki.miraheze.org` to `citywiki.eastsideurbanism.org` and `bunkshelf.miraheze.org` to `magebunkshelf.net`, both using the […] 13https://github.com/miraheze/ssl/pull/857#issuecomment-2954368034 [00:51:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749412280000&orgId=1&to=1749430310044 [00:52:12] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [00:53:08] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [00:54:08] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [01:01:36] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49859MiB (10% inode=99%); [01:06:10] PROBLEM - cp36 Nginx Backend for mw151 on cp36 is CRITICAL: connect to address localhost and port 8113: Connection refused [01:06:12] PROBLEM - cp36 Nginx Backend for mwtask181 on cp36 is CRITICAL: connect to address localhost and port 8160: Connection refused [01:06:13] PROBLEM - cp36 Nginx Backend for phorge171 on cp36 is CRITICAL: connect to address localhost and port 8202: Connection refused [01:06:13] PROBLEM - cp36 Nginx Backend for mwtask171 on cp36 is CRITICAL: connect to address localhost and port 8161: Connection refused [01:06:16] PROBLEM - cp36 Nginx Backend for puppet181 on cp36 is CRITICAL: connect to address localhost and port 8204: Connection refused [01:06:22] PROBLEM - cp36 Nginx Backend for reports171 on cp36 is CRITICAL: connect to address localhost and port 8205: Connection refused [01:06:24] PROBLEM - cp36 Nginx Backend for mw152 on cp36 is CRITICAL: connect to address localhost and port 8114: Connection refused [01:06:28] PROBLEM - cp36 Nginx Backend for mw183 on cp36 is CRITICAL: connect to address localhost and port 8127: Connection refused [01:06:31] PROBLEM - cp36 Nginx Backend for mwtask151 on cp36 is CRITICAL: connect to address localhost and port 8162: Connection refused [01:06:33] PROBLEM - cp36 Nginx Backend for matomo151 on cp36 is CRITICAL: connect to address localhost and port 8203: Connection refused [01:06:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 38.46.223.205/cpweb, 2602:294:0:b13::110/cpweb [01:06:52] PROBLEM - cp36 Nginx Backend for mw193 on cp36 is CRITICAL: connect to address localhost and port 8131: Connection refused [01:06:52] PROBLEM - cp36 Nginx Backend for mw161 on cp36 is CRITICAL: connect to address localhost and port 8115: Connection refused [01:06:54] PROBLEM - cp36 Nginx Backend for mw173 on cp36 is CRITICAL: connect to address localhost and port 8125: Connection refused [01:06:57] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [01:06:58] PROBLEM - cp36 Nginx Backend for mw153 on cp36 is CRITICAL: connect to address localhost and port 8121: Connection refused [01:07:02] PROBLEM - cp36 Nginx Backend for mw182 on cp36 is CRITICAL: connect to address localhost and port 8120: Connection refused [01:07:04] PROBLEM - cp36 Nginx Backend for mw191 on cp36 is CRITICAL: connect to address localhost and port 8129: Connection refused [01:07:07] PROBLEM - cp36 Nginx Backend for mw171 on cp36 is CRITICAL: connect to address localhost and port 8117: Connection refused [01:07:07] PROBLEM - cp36 Nginx Backend for mw172 on cp36 is CRITICAL: connect to address localhost and port 8118: Connection refused [01:07:59] PROBLEM - cp36 NTP time on cp36 is CRITICAL: NTP CRITICAL: Offset 0.7553577125 secs [01:08:06] RECOVERY - cp36 Nginx Backend for mw151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8113 [01:08:10] RECOVERY - cp36 Nginx Backend for mwtask181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8160 [01:08:11] RECOVERY - cp36 Nginx Backend for phorge171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8202 [01:08:11] RECOVERY - cp36 Nginx Backend for mwtask171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8161 [01:08:14] RECOVERY - cp36 Nginx Backend for puppet181 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8204 [01:08:17] RECOVERY - cp36 Nginx Backend for reports171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8205 [01:08:23] RECOVERY - cp36 Nginx Backend for mw152 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8114 [01:08:27] RECOVERY - cp36 Nginx Backend for mw183 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8127 [01:08:27] RECOVERY - cp36 Nginx Backend for mwtask151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8162 [01:08:30] RECOVERY - cp36 Nginx Backend for matomo151 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8203 [01:08:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:08:48] RECOVERY - cp36 Nginx Backend for mw161 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8115 [01:08:50] RECOVERY - cp36 Nginx Backend for mw173 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8125 [01:08:52] RECOVERY - cp36 Nginx Backend for mw193 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8131 [01:08:55] RECOVERY - cp36 Nginx Backend for mw153 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8121 [01:08:57] RECOVERY - cp36 Nginx Backend for mw182 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8120 [01:09:00] RECOVERY - cp36 Nginx Backend for mw191 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8129 [01:09:06] RECOVERY - cp36 Nginx Backend for mw171 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8117 [01:09:06] RECOVERY - cp36 Nginx Backend for mw172 on cp36 is OK: TCP OK - 0.000 second response time on localhost port 8118 [01:09:58] RECOVERY - cp36 NTP time on cp36 is OK: NTP OK: Offset 3.331899643e-05 secs [01:10:48] !log reboot cp36 [01:10:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:10:52] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [01:24:08] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 38% [01:32:08] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [01:34:38] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: connect to address 10.0.15.114 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [01:34:44] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.079 second response time [01:35:02] PROBLEM - cp38 Varnish Backends on cp38 is CRITICAL: 1 backends are down. mw151 [01:35:12] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw151 [01:35:15] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 31 backends are healthy [01:35:20] RECOVERY - cp38 Varnish Backends on cp38 is OK: All 31 backends are healthy [01:47:25] PROBLEM - cp36 Varnish Backends on cp36 is WARNING: No backends detected. If this is an error, see readme.txt [01:47:26] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [01:47:40] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [01:49:43] PROBLEM - cp36 Varnish Backends on cp36 is WARNING: No backends detected. If this is an error, see readme.txt [01:50:04] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 19 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw183 mw191 mw192 mw193 mw201 mw202 mw203 mediawiki [01:57:09] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 4088 bytes in 0.044 second response time [01:58:08] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 34% [02:00:05] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/958982c5797d3ea953a439a1d494d983a69edd42 [02:00:05] 02puppet/03main 07CosmicAlpha 03958982c Use fsslc.wtnet for backends [02:01:50] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 31 backends are healthy [02:12:40] PROBLEM - cloud20 IPMI Sensors on cloud20 is UNKNOWN: ID | Date | Time | Name | Type | State | Event1 | Jun-04-2025 | 21:35:06 | SEL | Event Logging Disabled | Nominal | Log Area Reset/Cleared2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 8 [02:12:40] 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 ipmi_sel_parse: internal IPMI error-> Execution of / [02:12:40] pmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [02:13:57] PROBLEM - cloud20 IPMI Sensors on cloud20 is WARNING: IPMI Status: Warning [114 system event log (SEL) entries present] [04:08:08] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 48% [04:16:07] @paladox @cosmicalpha is this anything to worry about? [04:16:17] Probably not [04:16:24] Not many wikis use cp36 [04:16:34] only like 20 [04:16:53] Only non-cloudflare uses cp36 [04:17:11] so wikitide.org subdomians and farthestfrontier.wiki and thats all [04:18:08] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 26% [04:18:21] And there it recovered @aeywoo [04:33:48] Interesting. [04:51:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749412280000&orgId=1&to=1749444710078 [05:06:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749412280000&orgId=1&to=1749445610080[Grafana] FIRING: The estimated time for the MediaWiki JobQueue to clear is excessively high (8 hours) for an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749441860000&orgId=1&to=1749445610080 [05:16:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749412280000&orgId=1&to=1749446210083[Grafana] RESOLVED: MediaWiki JobQueue is stalled https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749441860000&orgId=1&to=1749446180000 [08:58:39] PROBLEM - cloud20 IPMI Sensors on cloud20 is UNKNOWN: ID | Date | Time | Name | Type | State | Event1 | Jun-04-2025 | 21:35:06 | SEL | Event Logging Disabled | Nominal | Log Area Reset/Cleared2 3 4 5 ipmi_sel_parse: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sen [08:58:39] --sensor-types=all [09:00:39] PROBLEM - cloud20 IPMI Sensors on cloud20 is WARNING: IPMI Status: Warning [123 system event log (SEL) entries present] [09:16:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749412280000&orgId=1&to=1749460610116 [09:31:08] 123 SEL entries ain't good [09:32:21] https://grafana.wikitide.net/d/GtxbP1Xnk/mediawiki?from=2025-06-08T19:51:20.000Z&orgId=1&to=2025-06-09T09:16:50.116Z&timezone=browser&var-node=$__all&var-job=$__all&viewPanel=panel-57 is broken [09:41:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1749412280000&orgId=1&to=1749461830010 [12:11:56] [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/145be911e91c38c38a0f70526736e0d78e773348 [12:11:56] 02DiscordNotifications/03main 07translatewiki.net 03145be91 Localisation updates from https://translatewiki.net. [12:11:57] [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/b7c0a0681447fce980102a8ea629fba88d6d433e [12:11:57] 02landing/03main 07translatewiki.net 03b7c0a06 Localisation updates from https://translatewiki.net. [12:11:58] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/f41fec5ba6e36723252ac9eceba6ea12a9365e41 [12:12:00] 02MirahezeMagic/03main 07translatewiki.net 03f41fec5 Localisation updates from https://translatewiki.net. [12:12:02] [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/1b8e60990d426566dafd666d4dae6c7d4081ba73 [12:12:03] 02ManageWiki/03main 07translatewiki.net 031b8e609 Localisation updates from https://translatewiki.net. [12:12:49] miraheze/landing - translatewiki the build passed. [12:17:41] miraheze/ManageWiki - translatewiki the build passed. [12:26:31] miraheze/MirahezeMagic - translatewiki the build has errored. [12:31:32] miraheze/DiscordNotifications - translatewiki the build has errored. [13:34:57] !log recover 2 accounts by resetting the email following approval from T&S [13:34:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:02] [02ssl] 07SomeMWDev opened pull request #858: T13826: Redirect backroomszhwiki (07miraheze:03main...07SomeMWDev:03T13826) 13https://github.com/miraheze/ssl/pull/858 [13:57:23] [02ssl] 07RhinosF1 self-assigned pull request #858 (T13826: Redirect backroomszhwiki) 13https://github.com/miraheze/ssl/pull/858 [13:57:35] [02ssl] 07RhinosF1 approved pull request #858 13https://github.com/miraheze/ssl/pull/858#pullrequestreview-2910159421 [13:57:43] [02ssl] 07RhinosF1 merged 07SomeMWDev's pull request #858: T13826: Redirect backroomszhwiki (07miraheze:03main...07SomeMWDev:03T13826) 13https://github.com/miraheze/ssl/pull/858 [13:57:44] [02ssl] 07RhinosF1 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/beea172d3a986d5c4680b97316a8cd192fe2cc9e [13:57:46] 02ssl/03main 07SomeRandomDeveloper 03beea172 T13826: Redirect backroomszhwiki (#858) [13:59:49] [02ssl] 07coderabbitai[bot] commented on pull request #858: A new redirect entry named `backroomszhwiki` was added to the `redirects.yaml` configuration. This entry sets up a redirect from `backroomszh.miraheze.org` to `www.backroomszh.wiki` with specified SSL certificate and security settings. No other […] 13https://github.com/miraheze/ssl/pull/858#issuecomment-2955883435 [14:01:48] this has got to be a new record or sth lol, 7 minutes from task creation to PR merged [14:11:25] especially compared to the one that was open for 3 weeks ☠️😭 [14:12:39] i was looking at my laptop and bored [14:16:42] oh yeah I had that one on my todo but somehow forgot about it [14:16:56] ah [14:18:08] i forget about most things on my TODO [15:12:54] [02mw-config] 07OAuthority commented on pull request #5980: LGTM 13https://github.com/miraheze/mw-config/pull/5980#issuecomment-2956093948 [15:13:04] [02mw-config] 07OAuthority merged 07SomeMWDev's pull request #5980: T13789: Add wgPhonosInlineAudioPlayerMode (07miraheze:03main...07SomeMWDev:03T13789) 13https://github.com/miraheze/mw-config/pull/5980 [15:13:04] [02mw-config] 07OAuthority pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/e340645aa572e4585d58122e8442d371fffb118b [15:13:05] 02mw-config/03main 07SomeRandomDeveloper 03e340645 T13789: Add wgPhonosInlineAudioPlayerMode (#5980)… [15:13:57] miraheze/mw-config - OAuthority the build passed. [15:21:59] @originalauthority could you please pull the config for this commit? [15:33:24] !log [oa@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [15:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:33:46] in progress ^ [15:33:47] !log [oa@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [15:33:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:33:52] finito [15:34:50] thx [16:35:36] miraheze/puppet - Universal-Omega the build passed. [16:47:35] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/dd485ea594017be784559979985edd20abc6924f [16:47:35] 02puppet/03main 07CosmicAlpha 03dd485ea mwdeploy: just use Debug not full fqdn for location [16:49:13] miraheze/puppet - Universal-Omega the build passed. [17:29:45] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/cbbc595e83bcf9065a1e1ad111eecb55ee57bb1f [17:29:45] 02puppet/03main 07CosmicAlpha 03cbbc595 Add fsslc.wtnet versions for some nodes [17:36:15] [02MediaWikiDebugJS] 07Universal-Omega pushed 1 new commit to 03cleanup-content_script 13https://github.com/miraheze/MediaWikiDebugJS/commit/32c0bfcf8148fd16839ba86a2ed826f412cffe63 [17:36:15] 02MediaWikiDebugJS/03cleanup-content_script 07CosmicAlpha 0332c0bfc Merge branch 'main' into cleanup-content_script [17:54:59] PROBLEM - cp38 Disk Space on cp38 is CRITICAL: DISK CRITICAL - free space: / 27245MiB (5% inode=99%); [18:12:11] [02MediaWikiDebugJS] 07Universal-Omega pushed 1 new commit to 03cleanup-content_script 13https://github.com/miraheze/MediaWikiDebugJS/commit/5efe5d9720b6bcc10a29d7f7a7710f385d9229b8 [18:12:11] 02MediaWikiDebugJS/03cleanup-content_script 07CosmicAlpha 035efe5d9 Add eslint [18:12:37] [02MediaWikiDebugJS] 07Universal-Omega retitled PR #16: "Cleanup content_script" ➜ "Cleanup content_script and add eslint" 13https://github.com/miraheze/MediaWikiDebugJS/pull/16 [18:55:02] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27184MiB (5% inode=99%); [20:05:13] Inlet temperature flapping like crazy [20:05:27] well below critical though, when I checked last night [20:17:49] Oh [20:17:55] We should change the threshold then [21:21:14] PROBLEM - cloud15 IPMI Sensors on cloud15 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present] [21:39:07] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/35f749b3f4633fe781f07a9ed9e55f044f8ee50f [21:39:07] 02puppet/03main 07CosmicAlpha 0335f749b Remove wikitide.net version of phorge171 [21:39:18] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/916a2ad262a0e516ffabc39a3e90606b4923b019 [21:39:19] 02dns/03main 07CosmicAlpha 03916a2ad wikitide.net: remove alias for phorge171 [21:41:16] miraheze/dns - Universal-Omega the build passed. [21:47:15] adhd made me bored of my todo system [21:48:56] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/18b5d6ed1e71716107d6eb13b0ca3c464f28576f [21:48:56] 02puppet/03main 07CosmicAlpha 0318b5d6e Remove wikitide.net version of reports171 [21:49:17] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/1e06f16c4d8527ac30a83eabbb59d25f87d14521 [21:49:17] 02dns/03main 07CosmicAlpha 031e06f16 wikitide.net: remove alias for reports171 [21:50:24] miraheze/dns - Universal-Omega the build passed. [21:56:40] My to do list is more of a to not do list [21:58:26] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/4175e930fdc549e8f81c737f723af53681aa9391 [21:58:26] 02puppet/03main 07CosmicAlpha 034175e93 Remove wikitide.net version of graylog161 [22:00:17] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/60919d9db9325eeedfe9c0b257f823d9cd3310b6 [22:00:18] 02dns/03main 07CosmicAlpha 0360919d9 wikitide.net: remove alias for graylog161 [22:00:43] todo: review cargo's code once more [22:01:16] miraheze/dns - Universal-Omega the build passed. [22:03:53] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/49b2fa96d0e336da766c02bdb3d1cabb2f9672fd [22:03:54] 02puppet/03main 07CosmicAlpha 0349b2fa9 Add fsslc.wtnet node for matomo151 [22:03:54] [02ssl] 07asko1 opened pull request #859: T13829: Redirect fischwiki (07miraheze:03main...07asko1:03redirect-fisch) 13https://github.com/miraheze/ssl/pull/859 [22:04:47] [02ssl] 07Universal-Omega merged 07asko1's pull request #859: T13829: Redirect fischwiki (07miraheze:03main...07asko1:03redirect-fisch) 13https://github.com/miraheze/ssl/pull/859 [22:04:49] [02ssl] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/17efc6ee7321856a59bb5db5779f157057a379fc [22:04:50] 02ssl/03main 07Asko 0317efc6e T13829: Redirect fischwiki (#859)… [22:05:21] beat your record [22:05:45] https://issue-tracker.miraheze.org/T13829 [22:05:51] on your own task though [22:05:58] hey it still counts :p [22:06:42] https://cdn.discordapp.com/emojis/943096895545430087.webp?size=48 [22:09:16] [02ssl] 07coderabbitai[bot] commented on pull request #859: A new redirect entry named `fischwiki` was added to the `redirects.yaml` configuration. This entry specifies that requests to `fisch.miraheze.org` are redirected to `fischipedia.org` with defined SSL, certificate authority, and HSTS settings. No […] 13https://github.com/miraheze/ssl/pull/859#issuecomment-2957146917 [22:11:29] gonna have to watch the ssl backlog like a hawk [22:12:01] can't realte [22:12:06] relate* [22:13:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/595a0d27d0931c7a5dbd7f4b8bf512ec04c423cd [22:13:16] 02puppet/03main 07CosmicAlpha 03595a0d2 Remove wikitide.net version of matomo151 [22:13:28] realistically anyone who knows how to check if a domain works, copypaste, and create a pull request can do the redirect tasks [22:13:42] wdym? [22:15:01] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/548839311bc97e5753b830421ff53b8c0b8ffe0a [22:15:02] 02dns/03main 07CosmicAlpha 035488393 wikitide.net: remove alias for matomo151 [22:17:14] miraheze/dns - Universal-Omega the build passed. [22:22:42] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/6ae2b10e2a9b5a387f72eb9f0723a704c92973ee [22:22:42] 02puppet/03main 07CosmicAlpha 036ae2b10 mon181: update node [22:22:57] which part [22:28:43] !log mon181: restart ircecho as a test [22:28:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:29:07] brotato [22:33:02] asko1: the entire thing lol [22:33:15] no like which message lol [22:33:23] [10/06/2025 08:13] realistically anyone who knows how to check if a domain works, copypaste, and create a pull request can do the redirect tasks [22:33:30] ah that one [22:33:51] i mean those are all the qualifications i have to be making PRs to ssl/redirects.yaml [22:33:52] PROBLEM - db182 MariaDB Connections on db182 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_conne [22:33:52] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 PROBLEM - db182 SSH on db182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:43] PROBLEM - puppet181 SSH on puppet181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:55] PROBLEM - os151 SSH on os151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:00] uhoh? [22:35:01] PROBLEM - mem151 memcached on mem151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:11] PROBLEM - puppet181 WikiTideRenewSSL on puppet181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:34] PROBLEM - puppet181 puppetdb on puppet181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:07] PROBLEM - cloud19 SSH on cloud19 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:29] PROBLEM - mem151 SSH on mem151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:31] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:36:38] seems I broke the firewall allowing monitoring to connect I'm fixing but nothing is actually down. [22:36:46] PROBLEM - db182 Backups SQL icinga on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:36:57] PROBLEM - db182 Puppet on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:37:03] smoke but no fire, cool [22:37:13] PROBLEM - db182 Current Load on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:37:22] PROBLEM - db182 Disk Space on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:37:29] PROBLEM - db182 ferm_active on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:37:29] RECOVERY - puppet181 SSH on puppet181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u6 (protocol 2.0) [22:37:32] PROBLEM - puppet181 ferm_active on puppet181 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [22:37:40] PROBLEM - puppet181 Backups SSLKeys on puppet181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:37:40] PROBLEM - os151 Puppet on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:37:52] PROBLEM - puppet181 PowerDNS Recursor on puppet181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:37:55] RECOVERY - puppet181 WikiTideRenewSSL on puppet181 is OK: TCP OK - 0.000 second response time on 10.0.18.100 port 5000 [22:38:05] PROBLEM - os151 ferm_active on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:09] PROBLEM - os151 NTP time on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:23] PROBLEM - mem151 APT on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:29] PROBLEM - db182 conntrack_table_size on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:30] PROBLEM - db182 PowerDNS Recursor on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:30] PROBLEM - mem151 ferm_active on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:30] PROBLEM - db182 NTP time on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:41] RECOVERY - puppet181 puppetdb on puppet181 is OK: TCP OK - 0.000 second response time on 10.0.18.100 port 8081 [22:38:42] PROBLEM - db182 Backups SQL reports on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:53] PROBLEM - mem151 Disk Space on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:58] PROBLEM - db182 MariaDB on db182 is UNKNOWN: [22:38:58] PROBLEM - os151 conntrack_table_size on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:39:06] PROBLEM - os151 Current Load on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:39:13] PROBLEM - os151 Disk Space on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:39:19] PROBLEM - db182 APT on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:39:36] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [22:39:36] PROBLEM - cloud19 IPMI Sensors on cloud19 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:39:36] PROBLEM - db182 Backups SQL icingaweb2 on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:39:40] PROBLEM - os151 PowerDNS Recursor on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:39:42] PROBLEM - mem151 PowerDNS Recursor on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:39:55] PROBLEM - mem151 conntrack_table_size on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:39:57] > [10/06/2025 08:37] PROBLEM - db182 ferm_active on db182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:40:00] that's the fucking firewall [22:40:07] PROBLEM - cloud15 SSH on cloud15 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:17] PROBLEM - cloud19 PowerDNS Recursor on cloud19 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:40:29] CLOUD19 FIX THIS SHIT [22:40:39] PROBLEM - os151 APT on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:40:44] I BELIEVE IN YOU [22:40:49] OS151 YOU CAN DO THIS [22:40:51] PROBLEM - mem151 Puppet on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:40:51] PROBLEM - mem151 Current Load on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:40:51] PROBLEM - cloud19 APT on cloud19 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:40:55] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:40:56] MEM151 [22:40:57] MEM151 YESSS [22:41:00] PROBLEM - cloud19 Disk Space on cloud19 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:41:01] CLOUD15 TOO [22:41:01] YOU CAN DO THIS [22:41:03] PROBLEM - mem151 NTP time on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:41:06] PROBLEM - cloud15 Disk Space on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:41:08] wait no why am i being supportive [22:41:14] RECOVERY - puppet181 Backups SSLKeys on puppet181 is OK: FILE_AGE OK: /var/log/sslkeys-backup.log is 146456 seconds old and 92 bytes [22:41:19] PUPPET181!! [22:41:23] PROBLEM - cloud15 ferm_active on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:41:24] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:27] omg everything's getting fucking cooked [22:41:28] ah fuck [22:41:28] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:36] i don't even know what's going on [22:41:46] RECOVERY - puppet181 PowerDNS Recursor on puppet181 is OK: DNS OK: 0.043 seconds response time. puppet181.wikitide.net returns 10.0.18.100 [22:41:46] [10/06/2025 08:36] seems I broke the firewall allowing monitoring to connect I'm fixing but nothing is actually down. [22:41:52] PROBLEM - mw153 HTTPS on mw153 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [22:41:54] PROBLEM - test151 SSH on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:56] PROBLEM - cloud15 conntrack_table_size on cloud15 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:42:02] PROBLEM - cloud19 NTP time on cloud19 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:42:04] PROBLEM - cloud19 conntrack_table_size on cloud19 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:42:07] PROBLEM - test151 poolcounter process on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:42:08] PROBLEM - mwtask171 jobrunner.wikitide.net HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10003 milliseconds [22:42:12] PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:42:14] PROBLEM - mwtask171 jobrunner-high.wikitide.net HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9006: cURL returned 28 - Connection timed out after 10003 milliseconds [22:42:36] wow [22:42:43] icinga can't deal with this so much it just left [22:42:48] and on that note, i should leave too [22:43:19] I killed it [22:43:20] what the fuck just happened [22:43:33] girl i think everything crashed [22:43:42] Just me breaking firewalls [22:53:59] Welcome back icinga-miraheze [22:54:50] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/1cce2368a100c5925f5a0b2f4b8a85f9469230e2 [22:54:50] 02puppet/03main 07CosmicAlpha 031cce236 Remove wikitide.net version of mon181 [22:56:45] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/93e98cb49f69350f6e2e75aab61b4c4a0670a9e8 [22:56:45] 02dns/03main 07CosmicAlpha 0393e98cb wikitide.net: remove alias for mon181 [22:58:38] miraheze/dns - Universal-Omega the build passed. [22:58:54] "ircd? yeah, i rc deez nuts" [23:00:23] [Grafana] FIRING: Notification test https://grafana.wikitide.net/d/dashboard_uid?from=1749506423594&orgId=1&to=1749510023614 [23:00:43] great that works again too [23:00:43] siwwy bouncing grafanya :3 [23:00:50] dashboard not found lmfao [23:01:13] lol maybe the test notification is a fake dashboard? [23:06:59] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/03a686f998c58bae48c773886a1be1f0a1634bb8 [23:06:59] 02puppet/03main 07CosmicAlpha 0303a686f Add fsslc.wtnet versions of more nodes [23:08:55] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f8aa2c8c6044f96b27f9888cf75f1f190de15153 [23:08:56] 02puppet/03main 07CosmicAlpha 03f8aa2c8 Remove changeprop151 (no longer exists) [23:12:19] RECOVERY - mw153.fsslc.wtnet SSL Check on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 03 Apr 2040 12:14:00 PM GMT +0000. [23:15:10] wait, why does icinga-miraheze not have +v? [23:15:28] CosmicAlpha: ^ [23:15:57] It never did afaik [23:16:14] huh oka [23:16:15] y [23:18:39] Not sure [23:18:59] Maybe so it can flood less idk lol [23:19:14] I think there was a reason for it for a long time though but can't remember what it was... [23:22:44] pretty sure it used to need it due to ozone [23:24:42] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=meccatronicawiki --startOver (END - exit=0) [23:24:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:03] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=meccatronicawiki (END - exit=0) [23:25:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:39] [02mw-config] 07SomeMWDev opened pull request #5983: T13830: Add configuration for shintowiki (07miraheze:03main...07SomeMWDev:03T13830) 13https://github.com/miraheze/mw-config/pull/5983 [23:31:28] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/e0d254e641f725553d08ca88102db43adfafeea7 [23:31:29] 02mw-config/03main 07SomeRandomDeveloper 03e0d254e T13830: Add configuration for shintowiki (#5983)… [23:31:31] [02mw-config] 07Universal-Omega merged 07SomeMWDev's pull request #5983: T13830: Add configuration for shintowiki (07miraheze:03main...07SomeMWDev:03T13830) 13https://github.com/miraheze/mw-config/pull/5983 [23:31:32] miraheze/mw-config - SomeMWDev the build passed. [23:32:05] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [23:32:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:32:21] miraheze/mw-config - Universal-Omega the build passed. [23:32:26] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [23:32:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:18] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/3a1f79aa9e7c798ec0d074f1d88decd494225a8d [23:46:18] 02puppet/03main 07CosmicAlpha 033a1f79a Remove wikitide.net versions of nodes [23:49:44] [02dns] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/07536f292f57ee25c056400c2792e9c9efb5287d [23:49:44] 02dns/03main 07CosmicAlpha 0307536f2 wikitide.net: remove some aliases [23:50:15] CosmicAlpha: got comments on ? [23:52:07] miraheze/dns - Universal-Omega the build passed. [23:55:52] for now, i'll go manually fix said array [23:57:42] !log UPDATE mhglobal.mw_permissions SET perm_permissions = '[...]' WHERE perm_dbname = 'depressingdeathswiki' AND perm_group = 'owner'; [23:57:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:47] full command: UPDATE mhglobal.mw_permissions SET perm_permissions = '["","apihighlimits","applychangetags","autoconfirmed","autopatrol","block","blockemail","browsearchive","changetags","createaccount","createpage","createtalk","delete","deletechangetags","deletedhistory","deletedtext","deletelogentry","deleterevision","edit","editcontentmodel","editinterface","editprotected","editmyoptions","editmyprivateinfo","editmyusercss"," [23:57:48] editmyuserjson","editmyuserjs","editmywatchlist","editsemiprotected","editsitecss","editsitejson","editsitejs","editusercss","edituserjson","edituserjs","import","importupload","ipblock-exempt","managechangetags","markbotedits","mergehistory","minoredit","move","movefile","move-categorypages","move-rootuserpages","move-subpages","noratelimit","pagelang","patrol","patrolmarks","protect","purge","read","reupload","reupload-own"," [23:57:50] reupload-shared","rollback","sendemail","suppressredirect","unblockself","undelete","unwatchedpages","upload","upload_by_url","viewmyprivateinfo","viewmywatchlist","writeapi","abusefilter-modify","abusefilter-log-detail","abusefilter-view","abusefilter-log","abusefilter-modify-restricted","abusefilter-revert","abusefilter-view-private","abusefilter-log-private","override-antispoof","skipcaptcha","viewincidents","managewiki-core"," [23:57:51] managewiki-extensions","managewiki-settings","managewiki-namespaces","managewiki-permissions","noanalytics","nuke","ajaxpoll-vote","ajaxpoll-view-results","ajaxpoll-view-results-before-vote","comment","commentadmin","commentlinks","comment-delete-own","gadgets-edit","gadgets-definition-edit","replacetext","awardsmanage","giftadmin","view-dump","generate-dump","delete-dump","avatarremove","populate-user-profiles","","","",""]' WHERE [23:57:53]