[00:00:28] 10Beta-Cluster-Infrastructure: Request to help out - https://phabricator.wikimedia.org/T180757#3768618 (10greg) @Sau226 immediately is not a good time for me, unfortunately. PM me (I'm greg-g on Freenode) and we'll go from there. [00:01:12] 10Beta-Cluster-Infrastructure: Request to help out - https://phabricator.wikimedia.org/T180757#3768620 (10Sau226) Yes I will when I get time and then we can negotiate a new time if it isn't convenient. [00:01:12] @seen sau226 [00:09:48] 10Gerrit, 10Wikidata, 10User-Addshore: Move git repository of data-values/data-types PHP library out of mediawiki/extensions - https://phabricator.wikimedia.org/T180456#3758555 (10JeroenDeDauw) How about `wikibase/data-types` instead? Same for the package name. Reasoning: this package is not part of data-va... [00:12:10] 10Gerrit, 10Wikidata, 10User-Addshore: Move git repository of data-values/data-types PHP library out of mediawiki/extensions - https://phabricator.wikimedia.org/T180456#3768661 (10JeroenDeDauw) If I'm not mistaken we talked about moving the PHP code of that package directly into Wikibase.git two years ago or... [00:17:08] 10Beta-Cluster-Infrastructure, 10Performance-Team: Set up XHGui for Beta Cluster - https://phabricator.wikimedia.org/T180761#3768665 (10Krinkle) [00:18:58] 10Beta-Cluster-Infrastructure, 10Performance-Team: Set up XHGui for Beta Cluster - https://phabricator.wikimedia.org/T180761#3768689 (10Krinkle) [00:55:44] 10Beta-Cluster-Infrastructure: Make MediaWiki profiler in Beta match production - https://phabricator.wikimedia.org/T180766#3768748 (10Krinkle) [01:05:17] 10Beta-Cluster-Infrastructure, 10Performance-Team, 10Patch-For-Review: Make MediaWiki profiler in Beta match production - https://phabricator.wikimedia.org/T180766#3768805 (10Krinkle) [01:05:29] 10Beta-Cluster-Infrastructure, 10Performance-Team, 10Patch-For-Review: Make MediaWiki profiler in Beta match production - https://phabricator.wikimedia.org/T180766#3768748 (10Krinkle) [01:44:01] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [02:45:46] Project selenium-CirrusSearch » firefox,beta,Linux,BrowserTests build #589: 04FAILURE in 4 min 46 sec: https://integration.wikimedia.org/ci/job/selenium-CirrusSearch/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/589/ [04:07:41] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,BrowserTests build #580: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=BrowserTests/580/ [04:10:38] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #580: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/580/ [04:24:31] (03CR) 1020after4: [C: 032] Task Series scap plugin [tools/release] - 10https://gerrit.wikimedia.org/r/390429 (owner: 1020after4) [04:25:09] (03Merged) 10jenkins-bot: Task Series scap plugin [tools/release] - 10https://gerrit.wikimedia.org/r/390429 (owner: 1020after4) [04:30:32] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #580: 04FAILURE in 34 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/580/ [04:38:47] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Phabricator search hugely degraded in quality - https://phabricator.wikimedia.org/T180706#3767061 (10mmodell) [04:39:39] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Phabricator search hugely degraded in quality - https://phabricator.wikimedia.org/T180706#3768964 (10mmodell) [04:40:44] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Regression: Phabricator search hugely degraded in quality - https://phabricator.wikimedia.org/T180706#3767061 (10mmodell) [04:48:14] 10Beta-Cluster-Infrastructure, 10Performance-Team, 10Patch-For-Review: Make MediaWiki profiler in Beta match production - https://phabricator.wikimedia.org/T180766#3768984 (10Krinkle) After these patches it still isn't working. Compare ```name=test.wikimedia.beta $ curl -i 'https://test.wikimedia.beta.wmfl... [04:55:21] Project selenium-MultimediaViewer » chrome,beta,OS X 10.9,BrowserTests build #580: 04FAILURE in 59 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/580/ [06:49:02] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [06:54:23] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:29:22] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:54] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<10.00%) [07:40:05] 10Gerrit, 10Wikidata, 10User-Addshore: Move git repository of data-values/data-types PHP library out of mediawiki/extensions - https://phabricator.wikimedia.org/T180456#3769056 (10WMDE-leszek) 05Open>03Invalid I was also considering just moving it into Wikibase.git. Packagist shows a handful of uses outs... [07:40:11] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #547: 15ABORTED in 3 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/547/ [07:40:12] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #547: 15ABORTED in 3 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/547/ [08:51:57] I'm planning to merge https://gerrit.wikimedia.org/r/#/c/390377/ on Monday, that means that /etc/apt/sources.d/* files will only be managed by puppet [08:52:15] OIW locally created files will be yanked by puppet [08:52:40] currently deployment-prep uses no such files, but let me know if there are any concerns or objections [09:49:07] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline: Update Debian package for Blubber - https://phabricator.wikimedia.org/T179984#3769169 (10akosiaris) I 've tried building the package once more. It fails as it needs a tag for 00820cbd6bbcc98321c5a0d279394673425d0783 (I am gue... [09:49:09] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3769171 (10Addshore) [09:55:30] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3698328 (10Addshore) Once the train is unblocked (if it is) then the 2 patches in T180727#3769190 should be backported before it continues rolling forward. [09:59:27] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikimedia-Portals, and 2 others: Create a Jenkins Job that builds the portal deployment artifacts in CI - https://phabricator.wikimedia.org/T179694#3769202 (10Jdrewniak) >>! In T179694#3763562, @greg wrote: >>>! In T179694... [10:17:55] 10Gerrit, 10Wikidata, 10User-Addshore: Move git repository of data-values/data-types PHP library out of mediawiki/extensions - https://phabricator.wikimedia.org/T180456#3769229 (10Addshore) > Packagist shows a handful of uses outside of Wikibase. I think they actually just incorrectly list the package as a... [11:00:58] o/ - can I ask in here a quick github mirror set up for operations/software/druid_exporter ? [11:57:17] 10Continuous-Integration-Config, 10Dumps-Generation, 10Patch-For-Review: Add CI to all operations/dumps/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180328#3769359 (10hashar) [11:59:47] 10Continuous-Integration-Config, 10Dumps-Generation, 10Patch-For-Review: Add CI to all operations/dumps/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180328#3769383 (10hashar) > html: leave for now, it's not obsolete but nor do I have time to fix anything in there if it's brok... [12:00:10] 10Continuous-Integration-Config, 10Dumps-Generation, 10Patch-For-Review: Add CI to all operations/dumps/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180328#3769384 (10hashar) [12:11:50] 10Continuous-Integration-Config, 10Dumps-Generation, 10Patch-For-Review: Add CI to all operations/dumps/* repositories and archive obsolete ones - https://phabricator.wikimedia.org/T180328#3769408 (10hashar) [12:23:09] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:23:39] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #589: 04FAILURE in 1 min 37 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/589/ [12:46:25] 10Release-Engineering-Team (Kanban), 10Operations, 10Release Pipeline: Upgrade latest docker-registry.wikimedia.org/nodejs-devel to stretch - https://phabricator.wikimedia.org/T180524#3769460 (10MoritzMuehlenhoff) Yeah, I guess that would be an alternative to consider. [13:06:36] Project selenium-Math » chrome,beta,Linux,BrowserTests build #578: 04FAILURE in 2 min 35 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/578/ [13:54:39] 10Beta-Cluster-Infrastructure: Request to test centralauth operations on a test account - https://phabricator.wikimedia.org/T180757#3769571 (10Aklapper) [14:33:11] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #580: 04FAILURE in 1 min 10 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/580/ [15:03:58] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:05:24] PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:21:04] (03PS25) 10Umherirrender: Update tests for BlueSpice [integration/config] - 10https://gerrit.wikimedia.org/r/380790 [15:21:11] (03CR) 10Umherirrender: "Added BlueSpiceChecklist" [integration/config] - 10https://gerrit.wikimedia.org/r/380790 (owner: 10Umherirrender) [15:28:27] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #631: 04FAILURE in 6 min 27 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/631/ [15:28:43] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #631: 04FAILURE in 6 min 43 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/631/ [15:46:01] hi, I'd like to deploy a hotfix to fix an issue appearing on the search cluster [15:47:16] it's currently causing 1300 errors/min but could be worse while the rolling restart on eqiad continues ( elastic upgrade to 5.5.2 ) [15:48:08] the patch is pretty simple : https://gerrit.wikimedia.org/r/#/c/392057/ [15:48:34] the errors can be seen in logstash by searching for: CirrusSearch AND (i_o_exception OR illegal_state_exception) [15:50:12] for context: the error is due to a change in the binary protocol between nodes [15:50:30] the patch in mw-config disables this feature [15:50:43] greg-g: ^ [15:51:44] dcausse: +1 :) [15:52:12] hashar: thanks, deploying :) [15:52:12] dcausse: hopefully that solves the issue you are encountering :) [15:52:22] it is most probably easy to confirm it on the mwdebug hosts [15:52:27] I have +1ed the patch [15:52:41] hashar: sure [15:53:10] dcausse: ho and fatal monitor show a bunch of CirrusSearch notices [15:53:22] 214 Undefined index: _score in /srv/mediawiki/php-1.31.0-wmf.7/extensions/CirrusSearch/includes/Query/CompSuggestQueryBuilder.php on line 224 [15:53:22] 214 Undefined index: _id in /srv/mediawiki/php-1.31.0-wmf.7/extensions/CirrusSearch/includes/Query/CompSuggestQueryBuilder.php on line 223 [15:53:22] 30 Undefined index: 1 in /srv/mediawiki/php-1.31.0-wmf.8/includes/media/FormatMetadata.php on line 744 [15:53:22] :) [15:54:02] damn [15:54:06] ( wikipedias projects are still on wmf.7 ) [15:54:10] the train got hold yesterday [15:54:25] because of this one? [15:54:33] na for some other reason iirc [15:54:35] let me check [15:54:46] See: https://phabricator.wikimedia.org/T180714 "db1063 crashed" [15:54:47] ok I have 2 problems to fix now :) [15:54:52] so yeah hmm unrelated i believe [15:55:32] I guess you can start with the mediawiki-config change first [15:55:34] yes [15:57:15] (03PS2) 10Hashar: Add Daimona to trusted user list [integration/config] - 10https://gerrit.wikimedia.org/r/391812 (https://phabricator.wikimedia.org/T180683) (owner: 10Melos) [15:57:20] (03PS3) 10Hashar: Add Daimona to trusted user list [integration/config] - 10https://gerrit.wikimedia.org/r/391812 (https://phabricator.wikimedia.org/T180683) (owner: 10Melos) [15:57:36] (03CR) 10Hashar: Add Daimona to trusted user list (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/391812 (https://phabricator.wikimedia.org/T180683) (owner: 10Melos) [15:57:56] (03CR) 10Hashar: [C: 032] Add Daimona to trusted user list [integration/config] - 10https://gerrit.wikimedia.org/r/391812 (https://phabricator.wikimedia.org/T180683) (owner: 10Melos) [15:58:10] Yippee, build fixed! [15:58:11] Project selenium-MobileFrontend » chrome,beta,Linux,BrowserTests build #632: 09FIXED in 4 min 14 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/632/ [15:58:29] Yippee, build fixed! [15:58:29] Project selenium-MobileFrontend » firefox,beta,Linux,BrowserTests build #632: 09FIXED in 4 min 33 sec: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/632/ [15:59:12] (03Merged) 10jenkins-bot: Add Daimona to trusted user list [integration/config] - 10https://gerrit.wikimedia.org/r/391812 (https://phabricator.wikimedia.org/T180683) (owner: 10Melos) [16:07:20] hashar: first problem resolved, I'll a new task to deployment blockers, the notice in fatalmonitor will spam the logs badly if the next release reaches group2 [16:08:11] well done [16:11:07] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:12:28] dcausse: well thanks for the hotfix I guess. I am off myself :) [16:12:34] will check back later tonight [16:12:53] hashar: thanks! have a nice week-end! [16:14:28] danke [16:17:12] Project selenium-CentralNotice » chrome,beta,OS X 10.9,BrowserTests build #584: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/584/ [16:21:47] Project selenium-CentralNotice » firefox,beta,Windows 7,BrowserTests build #584: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=BrowserTests/584/ [16:23:50] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:26:34] twentyafterfour: I abandoned some cleanup you had put up for this and gutted the thing down the studs because that's all that is needed now https://gerrit.wikimedia.org/r/#/c/391969/ [16:28:43] 10Continuous-Integration-Config, 10MinusX: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3770178 (10Umherirrender) [16:31:37] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #202: 04FAILURE in 37 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/202/ [16:46:52] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikimedia-Portals, and 2 others: Create a Jenkins Job that builds the portal deployment artifacts in CI - https://phabricator.wikimedia.org/T179694#3770218 (10debt) [16:48:15] 10Release-Engineering-Team (Kanban), 10Discovery-Portal-Sprint: Create a dedicated deployment window for portal deployments - https://phabricator.wikimedia.org/T180401#3770229 (10debt) [16:53:34] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #202: 04FAILURE in 59 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/202/ [16:57:10] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:00:11] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:14:40] no_justification yay i got logstash working now. Needs a puppet change for this. But i see things todo with gerrit in logstash [17:14:48] http://gerrit-logstash.wmflabs.org/ [17:15:05] probaly need to tune that up so that it dosen't show unneeded stuff [17:15:17] thanks to ebernhardson who help me :) [17:15:56] This is using gelf? [17:16:00] nope [17:16:01] socket [17:16:14] Ah, yay. So don't need to redo everything [17:16:15] apparently gerrit has problems loading libs that it dosent use [17:16:24] yep [17:16:29] * paladox makes puppet change [17:17:51] yay [17:17:57] it shows when you stop gerrit too [17:18:06] Stopped Gerrit SSHD [17:18:36] i will do a second change to switch logstash on for gerrit. First patch is fixing it up to use the correct configuation. [17:21:46] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3770447 (10greg) @anomie... [17:21:55] i will also update our log4j configuation as we need to load our custom one [17:22:21] as we are using manly the defaults from upstream. Lets add those [17:22:44] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3770463 (10MarcoAurelio)... [17:24:48] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)): "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3770477 (10greg) [17:26:22] \o hi hi no_justification ! [17:29:23] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)): "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3770497 (10greg) I get the issue far more ofte... [17:34:21] aha no_justification better logs now [17:34:22] with mirroring upstream log4j configuation [17:34:41] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)): "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3770507 (10MarcoAurelio) Yep, me too. I'm usin... [17:36:10] * paladox dosen't know how upstream get it to go into error_log. [17:36:21] i will just mirror our version of error_log [17:37:10] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:52] 10Gerrit: Create Gerrit group "mcr-reviewers" to contain reviewers to be added to all MCR related patches - https://phabricator.wikimedia.org/T180815#3770542 (10Addshore) [17:40:09] 10Gerrit, 10User-Addshore: Create Gerrit group "mcr-reviewers" to contain reviewers to be added to all MCR related patches - https://phabricator.wikimedia.org/T180815#3770554 (10Addshore) [17:41:16] addshore: I'm sync'ing the two client fixes for you [17:41:21] No ETA on migrating wikis yet tho [17:41:34] no_justification: thanks! [17:41:41] migrating wikis? being finishing the train? [17:42:34] Yeah [17:43:13] okay! [17:43:44] getting all sites to .8 is the only step now before the final bit of build killing :0 [17:43:48] :) [17:43:57] But, if it doesnt happen then it doesnt happen! [17:44:17] last night was a rather unexpected evening of database explosion after all [17:44:53] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3770594 (10Addshore) [17:44:56] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3770595 (10Addshore) [17:46:31] addshore: likely monday for wmf.8 everywhere, but we'll see how the weekend goes [17:48:34] Okay! [17:53:45] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3770656 (10greg) The db crash issue is now resolved (see: https://wikitech.wikimedia.org/wiki/Incident_documentation/20171116-s5-dewiki-wikidata ). We will resume... [17:54:56] greg-g: I'm guessing monday evening for me? :) (regular train time)? [17:58:46] addshore: up to no_justification really :) [17:59:03] We could do wikidata that *morning* my time [17:59:11] That'd finish group1 [17:59:20] Then I could do group2 around noonish regular train time [17:59:41] Sounds good! [18:00:10] I guess I'll leave Monday to preparing my large chain of patches, and actually deploy them on tuesday [18:01:13] Mhhhm, It might even make sense to wait for 1 more train to roll before doing the final switches [18:01:33] otherwise a rollback from .8 to .7 would probably make things explode..... [18:01:55] let's reduce explosion factor [18:02:00] :) [18:02:02] Which of course is unlikely, but if we need to... [18:02:36] never say never! [18:02:40] indeed [18:02:48] * greg-g watched that movie recently [18:03:18] I might reschedule the final bits to the start of december then, so wait for all wikis to get onto .9 first :) [18:03:30] wait.. .10 [18:04:09] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:04:39] right, there won't be a .9 [18:04:52] (I mean, we could theoretically branch it, but we won't be deploying it) [18:05:13] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:27] no_justification https://gerrit.wikimedia.org/r/#/c/392079/ :). (Have not finished updating the patch yet but it should look like that). [18:05:28] oooh cool, I'll be in Berlin when killing the build finally, maybe we will get a cake [18:05:46] any excuse for a cake [18:05:49] need to add the params to gerrit.service to load the file + add some puppet code for the log files. [18:06:49] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3698328 (10Johan) @greg Will the train reach all wikis on Monday, or will it come to the last wikis later next week (like Tuesday or Wednesday)? [18:08:44] addshore: and then, no_justification can finally stop doing the train every week :) [18:09:00] ??? =o [18:10:34] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3770733 (10greg) @johan: wmf.8 is on group0+group1 right now: https://tools.wmflabs.org/versions/ . It (wmf.8) will go to group2 on Monday (so all wikis). There w... [18:12:06] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T178635#3770748 (10Johan) Thanks. [18:14:21] Cool, deploy calendar all updated :) [18:24:14] no_justification done :). Wondering could you give it a review as im not sure what i should chown logs/ as and same for the files please? :) [18:33:34] "November 17th 2017, 18:33:02.861 - - Unloading plugin gelf-gerrit, version aeb1fe1" [18:33:40] my test plugin that i just deleted [18:34:38] greg-g: I'm actually a fan :p [18:34:41] I like it [18:35:08] paladox: gerrit2:gerrit2 should own the logs/ directory [18:35:14] ah thanks [18:35:22] oh woops i meant [18:35:23] chmod [18:35:44] i've set /logs to [18:35:45] mode => '0444', [18:35:53] and the log files to [18:35:54] mode => '0644', [18:36:04] not sure if those are correct [18:36:52] 10Release-Engineering-Team (Watching / External), 10Epic, 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017): Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733#3770885 (10Anomie) [18:38:38] paladox: 0444 on a directory wouldnt be enough. to list a directory you need to execute (1) it. BUT puppet adds that automagically in the background. so listing it would work anyways.. but not writing [18:38:59] drwxr-xr-x 2 gerrit2 gerrit2 4096 Nov 17 18:22 logs [18:39:22] yea, so that x comes from puppet by default [18:39:47] is that what you are getting from your patch? [18:40:17] nope [18:40:26] i was meaning what should i set it too? [18:40:33] as puppet i presume will change it [18:41:01] ah [18:41:03] found it [18:41:03] chmod 755 logs [18:41:05] 0755 on the logs dir [18:41:08] https://chmodreverse.com [18:42:03] the rest is correct :) [18:42:11] paladox: stat -c %a yourdirectory [18:42:29] ah thanks [18:42:31] root@gerrit-test3:/var/lib/gerrit2/review_site/logs# stat -c %a ../logs [18:42:31] 755 [18:43:18] yea, and 0644 on the files seems normal [18:44:33] * paladox cherry picks the change [18:49:20] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)): "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3770909 (10Anomie) If you can reproduce it, pl... [19:05:39] tested cherry pick [19:08:01] 10Gerrit-Migration, 10Differential, 10Wikimedia Phabricator RfC: Pulling patches from Phabricator does not give consistent commit hashes - https://phabricator.wikimedia.org/T136#3770973 (10TerraCodes) [19:27:04] patch works :) [19:36:38] no_justification: I don't think you'll get complaints from mukunda or tyler if you finish out the year :) [19:36:47] no_justification: (we can talk on Monday) [20:01:44] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikimedia-Portals, and 2 others: Create a Jenkins Job that builds the portal deployment artifacts in CI - https://phabricator.wikimedia.org/T179694#3771123 (10greg) >>! In T179694#3769202, @Jdrewniak wrote: > Not sure. In... [20:26:02] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #15: 04FAILURE in 39 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/15/ [20:48:23] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #582: 04FAILURE in 7 min 22 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/582/ [20:53:32] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #582: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/582/ [21:05:06] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3771300 (10Paladox) A change was merged in 2.13 to hopefully fix this problem by looking at the db instead of the index. [21:23:36] greg-g: any objection to provisionally scheduling ReadingLists production deployment for week after thanksgiving? [21:23:59] tgr: as long as it's all good to go (I haven't been paying attention to it) [21:24:11] has been in beta for a while; probably needs some performance improvements, I'll get those done next week [21:24:18] will add details to the tasks [21:24:22] * greg-g nods [21:44:24] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)): "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3771432 (10Anomie) All I'm seeing in P6346 is... [22:02:32] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikimedia-Portals, and 2 others: Create a Jenkins Job that builds the portal deployment artifacts in CI - https://phabricator.wikimedia.org/T179694#3771461 (10debt) @Jdrewniak and @greg — I'll ask OIT to create an email l... [22:04:06] no_justification i wonder should we get the logstash change merged on monday & have logstash switch on for gerrit? :) [22:04:40] Project selenium-PageTriage » chrome,beta,Linux,BrowserTests build #579: 04FAILURE in 6 min 39 sec: https://integration.wikimedia.org/ci/job/selenium-PageTriage/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/579/ [22:07:00] paladox: Yeah we could do it monday or tuesday or something [22:07:07] ok thanks :) [22:08:35] Project selenium-PageTriage » firefox,beta,Linux,BrowserTests build #579: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/selenium-PageTriage/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/579/ [22:09:15] heh gerrit's home page is getting a new look http://gerrit-documentation.storage.googleapis.com/beta/index.html [22:27:49] Project selenium-CentralAuth » firefox,beta,Linux,BrowserTests build #585: 04FAILURE in 7 min 49 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/585/ [22:44:10] 10Gerrit, 10User-Addshore: Create Gerrit group "mcr-reviewers" to contain reviewers to be added to all MCR related patches - https://phabricator.wikimedia.org/T180815#3771544 (10demon) 05Open>03Resolved a:03demon [[ https://gerrit.wikimedia.org/r/#/admin/groups/1406,members | Done ]] [22:49:42] 10Release-Engineering-Team (Kanban), 10Cleanup, 10GitHub-Mirrors, 10Repository-Admins, and 2 others: Inactive extension-WikiGrok in Diffusion / Github - https://phabricator.wikimedia.org/T180847#3771552 (10greg) [23:13:34] 10Gerrit, 10Release-Engineering-Team (Kanban), 10User-Addshore: Create Gerrit group "mcr-reviewers" to contain reviewers to be added to all MCR related patches - https://phabricator.wikimedia.org/T180815#3771598 (10greg) [23:37:36] 10Release-Engineering-Team, 10MinervaNeue, 10Readers-Web-Backlog: Many MinervaNeue browser tests are failing intermittently but often on Chrome and Firefox - https://phabricator.wikimedia.org/T180828#3771645 (10Jdlrobson) @zeljkofilipin @hashar the failures are a little unusual. Did anything change with our... [23:44:05] (03PS1) 10Legoktm: Release 0.5.2 [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/392171 [23:44:50] (03CR) 10Legoktm: [C: 032] Release 0.5.2 [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/392171 (owner: 10Legoktm) [23:45:59] (03Merged) 10jenkins-bot: Release 0.5.2 [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/392171 (owner: 10Legoktm) [23:57:23] Yippee, build fixed! [23:57:24] Project selenium-CentralAuth » firefox,beta,Linux,BrowserTests build #587: 09FIXED in 2 min 7 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/587/