[06:21:45] <wikibugs>	 10Phabricator, 10Tools: Publicly log account bans made using the phab-ban tool - https://phabricator.wikimedia.org/T200856 (10DannyS712) Disabled https://phabricator.wikimedia.org/p/Vutting/ for vandalism
[12:32:40] <wikibugs>	 (03CR) 10Umherirrender: [C: 03+1] build: Updating mkdirp to 0.5.3 [integration/docroot] - 10https://gerrit.wikimedia.org/r/582493 (owner: 10Libraryupgrader)
[13:04:11] <wikibugs>	 (03PS1) 10Kosta Harlan: Provide support for setting default configuration in dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072)
[13:27:03] <wikibugs>	 (03PS2) 10Kosta Harlan: Provide support for setting default configuration in dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072)
[13:31:39] <wikibugs>	 (03PS3) 10Kosta Harlan: Provide support for setting default configuration in dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/582498 (https://phabricator.wikimedia.org/T247072)
[13:36:09] <wikibugs>	 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Gerrit account got messed up - https://phabricator.wikimedia.org/T245545 (10Soda) I am able to login via the web interface. Thanks a lot @thcipriani :smile:
[13:56:20] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Wikimedia-Logstash, and 2 others: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324 (10Aklapper) a:05Paladox→03None
[15:24:11] <wikibugs>	 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @AMuigai: Hi, I have added you. (Usual default disclaimer: Please follow [guidelines](https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#New_projects...
[15:26:54] <wikibugs>	 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @Rileych: Hmm. Your Phabricator account is linked against a [non-WMF SUL account](https://meta.wikimedia.org/wiki/Special:Log?page=User:CRiley%20(WMF)) so your statement is...
[16:48:58] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: (Service Check Timed Out)
[16:50:06] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:50:13] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:02:52] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-parsoid11 is CRITICAL: (Service Check Timed Out)
[17:03:31] <wikibugs>	 10Release-Engineering-Team, 10Analytics: wmfphab reverting github/wikimedia/KafkaSSE master to old commit - https://phabricator.wikimedia.org/T248170 (10Krenair) Fixed in https://phabricator.wikimedia.org/source/kafkasse/uri/view/16514/#10171 hopefully
[17:03:47] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93279 bytes in 0.989 second response time
[17:04:58] <shinken-wm>	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93742 bytes in 1.123 second response time
[17:05:05] <shinken-wm>	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52337 bytes in 1.091 second response time
[17:07:19] <wikibugs>	 10Release-Engineering-Team, 10Analytics: wmfphab reverting github/wikimedia/KafkaSSE master to old commit - https://phabricator.wikimedia.org/T248170 (10Krenair) 05Open→03Resolved a:03Krenair Swapped Phab URLs to read-only and changed the GitHub URL to Observe (i.e. mirror from GitHub to Diffusion).
[19:01:07] <shinken-wm>	 PROBLEM - Puppet staleness on integration-puppetmaster-02 is CRITICAL: (Service Check Timed Out)
[21:44:15] <wikibugs>	 10Project-Admins: Create a dedicated project tag for #Small-Wiki-Tool-Kits - https://phabricator.wikimedia.org/T247418 (10Aklapper) 05Open→03Resolved a:03Aklapper Created as #small-wiki-toolkits.
[21:56:34] <wikibugs>	 (03PS1) 10Umherirrender: [ConfigManager] Run quibble [integration/config] - 10https://gerrit.wikimedia.org/r/582546
[22:20:42] <wikibugs>	 (03CR) 10Awight: "Concise!" (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/550087 (owner: 10Robert Vogel)
[23:11:06] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:11:17] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:14:56] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:39:48] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93273 bytes in 0.960 second response time
[23:40:57] <shinken-wm>	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93698 bytes in 0.882 second response time
[23:41:04] <shinken-wm>	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52334 bytes in 1.000 second response time
[23:49:52] <wikibugs>	 10Beta-Cluster-Infrastructure: App server process on deployment-mediawiki-07 keeps breaking - https://phabricator.wikimedia.org/T248246 (10Krenair)
[23:54:20] <wikibugs>	 10Beta-Cluster-Infrastructure: "The database is read-only until replication lag decreases" when saving preferences on beta - https://phabricator.wikimedia.org/T247617 (10Krenair) replication looks up: ` root@deployment-db06:~# mysql -S $(find /tmp -name mysql.sock) -e 'show slave status \G' *********************...