[00:00:56] 10Continuous-Integration-Infrastructure: Easy to read visualization for PHPUnit report on Jenkins - https://phabricator.wikimedia.org/T116451#1749871 (10Mattflaschen) 3NEW [00:06:40] 10Beta-Cluster-Infrastructure: +Sysop for User:Mww113 - https://phabricator.wikimedia.org/T116364#1749878 (10Mww113) No worries. It's not really an urgent request. I won't actually need to do the tests until after the code is merged. I'm just requesting preemptively so I don't have to wait around for a sysop fla... [00:14:33] (03CR) 10Legoktm: [C: 032] Add Doxygen and test coverage for mediawiki/oauthclient-php [integration/config] - 10https://gerrit.wikimedia.org/r/248562 (owner: 10Gergő Tisza) [00:15:32] (03Merged) 10jenkins-bot: Add Doxygen and test coverage for mediawiki/oauthclient-php [integration/config] - 10https://gerrit.wikimedia.org/r/248562 (owner: 10Gergő Tisza) [00:15:43] !log deploying https://gerrit.wikimedia.org/r/248562 [00:15:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:18:15] (03PS1) 10Legoktm: Move doxygen/coverage for mediawiki/oauthclient-php into postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/248576 [00:18:25] (03CR) 10Legoktm: [C: 032] Move doxygen/coverage for mediawiki/oauthclient-php into postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/248576 (owner: 10Legoktm) [00:19:25] (03Merged) 10jenkins-bot: Move doxygen/coverage for mediawiki/oauthclient-php into postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/248576 (owner: 10Legoktm) [00:19:43] !log deploying https://gerrit.wikimedia.org/r/248576 [00:20:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:25:45] (03PS1) 10Legoktm: doc: Add oauthclient-php link [integration/docroot] - 10https://gerrit.wikimedia.org/r/248578 [00:26:21] (03CR) 10Legoktm: [C: 032] doc: Add oauthclient-php link [integration/docroot] - 10https://gerrit.wikimedia.org/r/248578 (owner: 10Legoktm) [00:26:35] (03Merged) 10jenkins-bot: doc: Add oauthclient-php link [integration/docroot] - 10https://gerrit.wikimedia.org/r/248578 (owner: 10Legoktm) [00:35:39] James_F|Away: https://www.mediawiki.org/wiki/User:Legoktm/library_upgrader add your ideas please [00:46:29] PROBLEM - Free space - all mounts on deployment-db2 is CRITICAL: CRITICAL: deployment-prep.deployment-db2.diskspace._mnt.byte_percentfree (<11.11%) [01:02:47] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:48] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:48] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:27] * Krenair grumbles [01:06:37] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 38968 bytes in 0.476 second response time [01:06:42] also this: PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:47] I wonder what happened [01:06:53] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 38645 bytes in 1.218 second response time [01:06:59] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 38647 bytes in 1.585 second response time [01:21:09] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:10] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:20] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:20] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:21] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 38658 bytes in 0.515 second response time [01:24:21] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:37] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 30410 bytes in 0.582 second response time [01:55:08] 10Beta-Cluster-Infrastructure: +Sysop for User:Mww113 - https://phabricator.wikimedia.org/T116364#1749948 (10MaxSem) Aslo, can we see the actual commit? [02:05:01] PROBLEM - Free space - all mounts on deployment-db2 is CRITICAL: CRITICAL: deployment-prep.deployment-db2.diskspace._mnt.byte_percentfree (<33.33%) [02:09:39] marxarelli, how's that restore coming along? [02:10:02] Krenair: keeps coughing on 'ERROR 1146 (42S02) at line 103829: Table 'deploymentwiki.hitcounter' doesn't exist' [02:10:13] and breaking entirely? [02:10:40] mmmm [02:10:55] it might have finished the schema restore otherwise [02:14:41] i'll go ahead and try restoring the data and see what happens [02:15:03] !log restoring data on deployment-db2 [02:15:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:20:26] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:20:51] Krenair: i think it just choked on another broken view in that funky labswiki db. otherwise, it seems to be going ok [02:21:23] * marxarelli goes to eat his dinner while this runs [03:12:51] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #858: 04FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/858/ [03:16:37] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #758: 04FAILURE in 26 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/758/ [03:18:28] 10Continuous-Integration-Infrastructure: Easy to read visualization for PHPUnit report on Jenkins - https://phabricator.wikimedia.org/T116451#1750034 (10Mattflaschen) @legoktm showed me where this is. E.g. https://integration.wikimedia.org/ci/job/mediawiki-extensions-zend/24956/testReport/%28root%29/ [03:18:28] !log finished restoring data on deployment-db2. replication is working once again [03:18:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:18:36] 10Continuous-Integration-Infrastructure: Easy to read visualization for PHPUnit report on Jenkins - https://phabricator.wikimedia.org/T116451#1750036 (10Mattflaschen) 5Open>3Invalid [03:32:28] RECOVERY - Free space - all mounts on deployment-db2 is OK: OK: All targets OK [03:36:34] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36559 bytes in 1.195 second response time [03:36:52] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 36253 bytes in 0.477 second response time [03:36:58] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 36271 bytes in 0.482 second response time [03:43:00] marxarelli, looks like it broke again [03:54:43] !log restoring deployment-db2 again ... [03:54:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:57:05] Krenair: found and removed the failed labswiki view from the schema dump and it restored cleanly this time. data is in progress [04:03:25] 10Beta-Cluster-Infrastructure: +Sysop for User:Mww113 - https://phabricator.wikimedia.org/T116364#1750064 (10Mww113) https://gerrit.wikimedia.org/r/#/c/248593/ [04:18:26] PROBLEM - Free space - all mounts on deployment-db2 is CRITICAL: CRITICAL: deployment-prep.deployment-db2.diskspace._mnt.byte_percentfree (<55.56%) [04:48:23] RECOVERY - Free space - all mounts on deployment-db2 is OK: OK: All targets OK [04:53:06] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 37507 bytes in 0.745 second response time [05:17:16] PROBLEM - Puppet failure on deployment-cache-parsoid04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:58:28] !log deployment-db2 data restored, replication working [05:58:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [06:01:20] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Beta Cluster outage: deployment-db2 disk filled up, locked db replication - https://phabricator.wikimedia.org/T116447#1750087 (10dduvall) 5Open>3Resolved The corrupt binlog was likely a result of `/mnt` filling up due to an earlier long/massive quer... [06:22:25] Yippee, build fixed! [06:22:25] Project beta-update-databases-eqiad build #3868: 09FIXED in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/3868/ [06:25:01] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 10Wikimedia-Extension-setup, 5Patch-For-Review: Set up UrlShortener extension on the beta cluster - https://phabricator.wikimedia.org/T116444#1750094 (10Legoktm) [06:46:33] Yippee, build fixed! [06:46:33] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #2833: 09FIXED in 32 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/2833/ [07:09:07] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [07:14:03] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:07:17] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 10Wikimedia-Extension-setup, 5Patch-For-Review: Set up UrlShortener extension on the beta cluster - https://phabricator.wikimedia.org/T116444#1750132 (10Legoktm) * {346d522f1cb645204b3f01963f97bee2485832df} * {13c489b484dc2e17d8591b8ea902d... [08:07:45] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-UrlShortener, 10Wikimedia-Extension-setup, 5Patch-For-Review: Set up UrlShortener extension on the beta cluster - https://phabricator.wikimedia.org/T116444#1749684 (10Legoktm) [08:32:13] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #761: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/761/ [09:16:45] 10Continuous-Integration-Config, 7I18n: Configure banana checker for i18n files to run on all MediaWiki extensions and skins (tracking) - https://phabricator.wikimedia.org/T94547#1750204 (10Umherirrender) [09:20:46] 10Continuous-Integration-Config, 7I18n: Configure banana checker for i18n files to run on all MediaWiki extensions and skins (tracking) - https://phabricator.wikimedia.org/T94547#1750212 (10Umherirrender) [09:30:36] PROBLEM - Puppet failure on deployment-fluorine is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:06:51] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Beta Cluster outage: deployment-db2 disk filled up, locked db replication - https://phabricator.wikimedia.org/T116447#1750221 (10jcrespo) > Good, that this happens only at beta. Imagine that would happend at production.... I think this could not disrup... [12:50:31] is beta now stable enough, to delete not a few pages? [14:10:06] 10Beta-Cluster-Infrastructure: +Sysop for User:Mww113 - https://phabricator.wikimedia.org/T116364#1750492 (10Luke081515) 5stalled>3Open Ok, problem at beta is now fixed, to confirm: * Sysop at enwiki and metawiki beta to edit antispoof related messages Anyone against this? Otherwise I would assign the rights... [14:11:00] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 6Discovery: Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#1750498 (10Luke081515) [14:16:12] Is the last thing a bug? I don't make anything at that task [14:35:47] Yippee, build fixed! [14:35:47] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #302: 09FIXED in 7 min 46 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/302/ [15:08:41] (03PS5) 10Paladox: Convert 'operations-puppet-doc' job to run on a labs slave [integration/config] - 10https://gerrit.wikimedia.org/r/204982 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [15:08:57] (03CR) 10Paladox: "Rebased." [integration/config] - 10https://gerrit.wikimedia.org/r/204982 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [15:20:03] (03PS2) 10Paladox: Use generic qunit job for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/208893 (owner: 10Legoktm) [15:20:16] (03CR) 10Paladox: "Rebased." [integration/config] - 10https://gerrit.wikimedia.org/r/208893 (owner: 10Legoktm) [15:20:22] (03CR) 10Paladox: [C: 031] Use generic qunit job for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/208893 (owner: 10Legoktm) [15:26:21] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/198185 (owner: 10Legoktm) [16:01:17] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Beta Cluster outage: deployment-db2 disk filled up, locked db replication - https://phabricator.wikimedia.org/T116447#1750642 (10greg) Thanks @jcrespo, would it be useful to have a conversation about what you're thinking about specifically? We (RelEng)... [16:03:17] Luke081515: re deleting some pages. why not, see what else we can break ;) [16:03:31] Luke081515: more seriously, be safe first! :) [16:04:49] :) [16:11:39] 10Browser-Tests, 10MediaWiki-extensions-GettingStarted, 5Patch-For-Review: Upgrade GettingStarted browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99655#1750646 (10zeljkofilipin) [16:11:43] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: [Task] investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1750648 (10zeljkofilipin) [16:11:48] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow, 10MediaWiki-extensions-GettingStarted, and 4 others: undefined method `last_session_ids=' for MediawikiSelenium::BrowserFactory::Chrome:Class (NoMethodError) - https://phabricator.wikimedia.org/T114368#1750645 (10zeljkofilipin) 5Open>3Resolved [16:25:10] 10MediaWiki-Codesniffer, 5Patch-For-Review: Add sniff for "if/while ( $a = foo() )" constructs in phpcs - https://phabricator.wikimedia.org/T92744#1750657 (10Physikerwelt) I'm wondering how to avoid assignment in while loops completely see for example http://php.net/manual/en/function.fgetcsv.php [18:30:46] Yippee, build fixed! [18:30:46] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #480: 09FIXED in 45 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/480/ [18:53:38] Yippee, build fixed! [18:53:39] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #481: 09FIXED in 38 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/481/ [20:21:06] Luke081515, hey [20:21:16] hey [20:21:18] per suggestion of legoktm in -operations, I've started the process to create a new beta wiki: https://gerrit.wikimedia.org/r/248639 [20:21:33] Great :) [20:23:06] Krenair: The first wikivoyage :) [20:23:37] YES [20:23:38] yes* [20:24:51] I'm going to wait for greg to comment before I hit the button, but I wanted to warn you in advance [20:25:55] ok, thank you :) [21:12:05] 10Beta-Cluster-Infrastructure: [[wikitech:]] in Beta should not link to non-existant wikitech.wikimedia.deployment.wmflabs.org - https://phabricator.wikimedia.org/T103248#1751065 (10Krinkle) 5Open>3Resolved a:3Krinkle [21:56:43] 10Beta-Cluster-Infrastructure, 6Labs: Figure out why wikipedia requires an extra DNS entry that the other sites do not - https://phabricator.wikimedia.org/T111661#1751131 (10Krenair) a:5Andrew>3Krenair [21:59:17] 10Beta-Cluster-Infrastructure, 6Labs: beta-hhvm.wmflabs.org? - https://phabricator.wikimedia.org/T111657#1751137 (10Krenair) Have also got cloudadmin and killed these entries in the labs global DNS config: ```beta-hhvm: beta-hhvm.wmflabs.org wikipedia-beta-hhvm: wikipedia.beta-hhvm.wmflabs.org``` [22:02:36] 10Beta-Cluster-Infrastructure, 6Labs: Figure out why wikipedia requires an extra DNS entry that the other sites do not - https://phabricator.wikimedia.org/T111661#1751138 (10Krenair) 5Open>3Resolved I got cloudadmin, removed the weird NovaAddress entry, and then removed the `wikipedia-beta: wikipedia.beta.... [23:10:04] Yippee, build fixed! [23:10:04] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #303: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/303/