[00:01:11] (03CR) 10Legoktm: "* These jobs should only run under python3...do I need to specify that somewhere?" [integration/config] - 10https://gerrit.wikimedia.org/r/175607 (owner: 10Legoktm) [00:08:42] 3MediaWiki-extensions-GettingStarted, Beta-Cluster: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#783128 (10greg) [00:08:49] 3MediaWiki-extensions-GettingStarted, Beta-Cluster: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#563334 (10greg) [00:08:54] 3Beta-Cluster: monitor unsigned salt keys - https://phabricator.wikimedia.org/T72862#783132 (10greg) [00:09:00] 3Beta-Cluster: beta: set $wgCategoryCollation for languages - https://phabricator.wikimedia.org/T47525#783134 (10greg) [00:09:37] 3Beta-Cluster: Make deployment prep have continuous replication lag - https://phabricator.wikimedia.org/T59583#783137 (10greg) [00:11:57] ok, enough phab-phun for today [00:11:58] :) [00:12:56] * greg-g just took a stab at the Beta Cluster board.... did some weird things, hopefully I didn't annoy antoine too much :) [00:12:57] greg-g: you misspelled "spam" [00:13:03] yeah, that :) [00:13:21] first few days will be a ton a spam [00:13:38] protip: go change your email settings to not send you changes to subscribers and other useless notifications [00:14:12] today was a first pass at #beta-cluster, tomorrow will be #ci [00:14:44] * bd808 has email dialed down to only status and comments [00:15:07] good idea [00:15:26] I like the firehose, but I know that's not everyone's style :P [00:15:35] (except cc changes) [00:26:48] greg-g: do you think we'll be scheduling a RelEngTeam thingie around the Hackathon? I see Rachel just announced that officially. [00:41:02] 3VisualEditor, Beta-Cluster: Beta Cluster: API PrefixSearch is taking a very long time to return, and returns nothing when it does - https://phabricator.wikimedia.org/T74332#783169 (10Jdforrester-WMF) [00:59:24] 3Beta-Cluster: Mobile redirect goes to wrong domain name on beta labs - https://phabricator.wikimedia.org/T73079#783202 (10kaldari) [01:03:13] 3MediaWiki-extensions-GettingStarted, Beta-Cluster: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#783205 (10faidon) [01:03:51] 3Beta-Cluster: Need a way to simulate replication lag to test replag issues - https://phabricator.wikimedia.org/T40945#783206 (10faidon) [02:31:09] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #307: FAILURE in 28 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/307/ [02:52:00] (03CR) 10Krinkle: Job template to run composer scripts (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/174410 (owner: 10Hashar) [02:53:44] (03CR) 10Krinkle: Job template to run composer scripts (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/174410 (owner: 10Hashar) [03:02:07] Yippee, build fixed! [03:02:07] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #301: FIXED in 30 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/301/ [03:21:26] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #186: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/186/ [03:21:59] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#783313 (10Jdforrester-WMF) [03:22:12] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#783315 (10Jdforrester-WMF) [03:35:17] Project beta-scap-eqiad build #31163: FAILURE in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31163/ [03:37:59] Yippee, build fixed! [03:37:59] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #96: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/96/ [03:55:12] Yippee, build fixed! [03:55:13] Project beta-scap-eqiad build #31165: FIXED in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31165/ [03:57:03] Yippee, build fixed! [03:57:04] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #150: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/150/ [04:37:22] !log restarted jenkins at 20:31 [04:37:25] Logged the message, Master [04:39:04] it's back up but i still can't auth [08:20:20] 3Beta-Cluster: Setup a mediawiki03 (or what not) on Beta Cluster that we can direct the security scanning work to - https://phabricator.wikimedia.org/T72181#783612 (10Qgil) [08:24:07] (03PS1) 10Gilles: TMH now passes jslint [integration/config] - 10https://gerrit.wikimedia.org/r/175668 [09:09:48] zeljkof: coming soon [09:15:27] aharoni: ping me when you are ready [09:25:32] zeljkof: READY [09:30:30] aharoni: in the hangout [09:41:42] 3Release-Engineering: Submit "A Self-Translating User Manual" talk to a couple of FOSDEM devrooms - https://phabricator.wikimedia.org/T1389#783775 (10zeljkofilipin) Ask Vikas if he would like to participate [09:55:25] Project beta-scap-eqiad build #31200: FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31200/ [10:13:03] (03CR) 10Hashar: [C: 032] "That is awesome. Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/175668 (owner: 10Gilles) [10:13:45] (03Merged) 10jenkins-bot: TMH now passes jslint [integration/config] - 10https://gerrit.wikimedia.org/r/175668 (owner: 10Gilles) [10:15:24] Yippee, build fixed! [10:15:24] Project beta-scap-eqiad build #31202: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31202/ [10:33:10] 3Beta-Cluster: Setup monitoring for Beta cluster - https://phabricator.wikimedia.org/T53497#783882 (10yuvipanda) @hashar We can tweak the body, indeed. Let me know what you would like :) [10:38:01] 3CirrusSearch, Beta-Cluster: Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#783896 (10hashar) [10:47:10] 3CirrusSearch, Beta-Cluster: Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#783908 (10hashar) Ganglia has been phased out in favor of using `diamond` on the hosts which collect host metrics and emit them to a labs Graphite: https://graphite.wmflabs.org/ . The 1 minute loa... [10:47:45] 3CirrusSearch, Beta-Cluster: Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#783909 (10hashar) [10:49:30] 3CirrusSearch, Beta-Cluster: Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#749029 (10hashar) [10:57:20] 3Beta-Cluster: hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - https://phabricator.wikimedia.org/T71979#783917 (10hashar) [10:57:47] 3Beta-Cluster: hhvm creates core file in /tmp/ filling mediawiki02 labs instance root partition - https://phabricator.wikimedia.org/T71979#783919 (10hashar) [11:00:58] 3Beta-Cluster: monitor unsigned salt keys - https://phabricator.wikimedia.org/T72862#783923 (10hashar) [11:02:56] 3Beta-Cluster: Setup monitoring for Beta cluster - https://phabricator.wikimedia.org/T53497#783926 (10faidon) [11:03:37] 3Release-Engineering, MediaWiki-Core-Team: Make sure scap rsync proxies aren't in jobrunner pool - https://phabricator.wikimedia.org/T1342#783927 (10Reedy) It's something that @bd808 has mentioned a few times, based on the logs, stuff using the job runners take longer IIRC. This can be then exacerbated by many... [11:12:06] 3Release-Engineering: Add icinga output for Beta Cluster to -operations channel - https://phabricator.wikimedia.org/T1334#783943 (10yuvipanda) poke [11:19:01] 3Release-Engineering: Add icinga output for Beta Cluster to -operations channel - https://phabricator.wikimedia.org/T1334#783944 (10hashar) I guess you will want to discuss with the rest of the operations team, I am not sure they should be diverted from production issues by beta cluster notifications. That bein... [11:21:47] 3Release-Engineering: Add icinga output for Beta Cluster to -operations channel - https://phabricator.wikimedia.org/T1334#783947 (10yuvipanda) Almost all puppet issues on deployment-prep are caused by changes to puppet that didn't take into account beta. Perhaps we could only relay puppet errors to -operations? [11:32:26] 3Release-Engineering: Convert Beta Cluster specific puppet configs to use Hiera - https://phabricator.wikimedia.org/T451#783957 (10yuvipanda) There's some documentation at https://wikitech.wikimedia.org/wiki/Puppet_Hiera [12:10:20] lubaochuan: could you send me the link to the hangout [12:10:26] ? [12:10:37] google plus link does not open anything [12:10:41] zeljkof: https://plus.google.com/hangouts/_/hoaevent/AP36tYffKZTdc1Y0eLrw66UXySb6ceZVb9p12KzZ7A-s7xN4MG19rA [12:11:29] 3Beta-Cluster: Setup monitoring for Beta cluster - https://phabricator.wikimedia.org/T53497#783995 (10yuvipanda) [12:11:30] 3Beta-Cluster: Set up graphite monitoring for the beta cluster - https://phabricator.wikimedia.org/T54357#783992 (10yuvipanda) [12:11:31] 3Beta-Cluster: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#783996 (10yuvipanda) [12:17:54] lubaochuan: https://github.com/zeljkofilipin/dotfiles/blob/master/.vimrc [12:22:37] lubaochuan: https://github.com/zeljkofilipin/dotfiles.git [12:26:49] lubaochuan: http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page [12:31:07] 3Release-Engineering, MediaWiki-Core-Team: Make sure scap rsync proxies aren't in jobrunner pool - https://phabricator.wikimedia.org/T1342#784056 (10Reedy) [12:34:00] 3Beta-Cluster: Subscribe shinken@shinken-01.eqiad.wmflabs to beta cluster alerts mailing list - https://phabricator.wikimedia.org/T75866#784057 (10yuvipanda) [12:35:51] 3Release-Engineering, MediaWiki-Core-Team: Make sure scap rsync proxies aren't in jobrunner pool - https://phabricator.wikimedia.org/T1342#784074 (10Joe) I have a patch that will make this easier to do: https://gerrit.wikimedia.org/r/#/c/174664/ Working on it today. [12:38:52] 3Beta-Cluster: Subscribe shinken@shinken-01.eqiad.wmflabs to beta cluster alerts mailing list - https://phabricator.wikimedia.org/T75866#784083 (10yuvipanda) p:5Triage>3Normal [12:52:27] (03PS1) 10Stan: split function to reduce cyclomatic complexity (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175686 [12:52:29] (03PS1) 10Stan: module/class doc comments (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175687 [12:52:31] (03PS1) 10Stan: use standard regex syntax (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175688 [12:52:33] (03PS1) 10Stan: remove rubocop todo [ruby/api] - 10https://gerrit.wikimedia.org/r/175689 [12:52:35] (03PS1) 10Stan: correct cop name [ruby/api] - 10https://gerrit.wikimedia.org/r/175690 [13:27:31] 3Beta-Cluster: Puppet fails on deployment-sca01 - https://phabricator.wikimedia.org/T75508#784144 (10hashar) [13:27:41] 3Beta-Cluster: Puppet fails on deployment-sca01 - https://phabricator.wikimedia.org/T75508#784145 (10hashar) 5Open>3Resolved a:3hashar The alert above was due to me revoking some puppet certificates while attempting to fix one instance. The cert is fine now and I have confirmed puppet run fine on the inst... [13:30:07] 3Beta-Cluster: Puppet failures on deployment-pdf01 - https://phabricator.wikimedia.org/T75506#784150 (10hashar) [13:30:47] 3Beta-Cluster: Puppet failures on deployment-pdf01 Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol - https://phabricator.wikimedia.org/T75506#784152 (10hashar) [13:31:47] 3Beta-Cluster: Puppet failures on deployment-pdf01 Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol - https://phabricator.wikimedia.org/T75506#754252 (10hashar) [13:32:42] 3Beta-Cluster: Puppet failures on deployment-pdf01 Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol - https://phabricator.wikimedia.org/T75506#784157 (10hashar) [13:33:59] 3Beta-Cluster: Puppet failures on deployment-bastion - https://phabricator.wikimedia.org/T75520#784158 (10hashar) [13:34:31] 3Beta-Cluster: Puppet failures on deployment-pdf01 Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol - https://phabricator.wikimedia.org/T75506#754252 (10hashar) [13:37:38] 3Beta-Cluster: Puppet failure on instance udplog due to udp2log class depends on ganglia/gmond - https://phabricator.wikimedia.org/T75516#784166 (10hashar) [13:37:58] 3Beta-Cluster: Puppet failure on instance udplog due to udp2log class depends on ganglia/gmond - https://phabricator.wikimedia.org/T75516#755216 (10hashar) [13:40:18] 3Beta-Cluster: hhvm apache fills /var/log/apache2 with access logs - https://phabricator.wikimedia.org/T75262#784171 (10hashar) [13:40:33] 3Beta-Cluster: hhvm apache fills /var/log/apache2 with access logs - https://phabricator.wikimedia.org/T75262#755023 (10hashar) [13:44:26] 3Beta-Cluster: Diamond logstash monitor fills /var/log/apache2 access log - https://phabricator.wikimedia.org/T74175#784177 (10hashar) [13:45:14] 3MediaWiki-extensions-PoolCounter, Beta-Cluster: Install and configure pool counter on beta cluster - https://phabricator.wikimedia.org/T72940#784181 (10hashar) [13:51:36] 3MediaWiki-General-or-Unknown, Beta-Cluster: UserMailer silently fails to deliver anything on betacluster - https://phabricator.wikimedia.org/T75736#784196 (1001tonythomas) [13:55:34] 3Release-Engineering, Beta-Cluster: Convert Beta Cluster specific puppet configs to use Hiera (tracking) - https://phabricator.wikimedia.org/T451#784207 (10hashar) [13:59:02] 3Beta-Cluster: Set up graphite monitoring for the beta cluster - https://phabricator.wikimedia.org/T54357#784222 (10hashar) +1 for the monitoring part of it. I am wondering what to do for MediaWiki profiling though. Should we use the same graphite instance or a dedicated one? I don't mind filing another task. [13:59:27] 3Release-Engineering, Beta-Cluster: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#784223 (10hashar) [14:11:38] 3Beta-Cluster: Set up graphite monitoring for the beta cluster - https://phabricator.wikimedia.org/T54357#784281 (10yuvipanda) Let's use the same one. That's how it is in prod. [14:16:57] 3Beta-Cluster: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#784292 (10hashar) [14:18:23] 3Beta-Cluster: Setup Gdash for beta cluster - https://phabricator.wikimedia.org/T75882 (10hashar) 3NEW p:3Normal [14:18:32] 3Beta-Cluster: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#784292 (10hashar) [14:19:07] 3Beta-Cluster: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#784321 (10hashar) [14:19:09] 3Beta-Cluster: Setup Gdash for beta cluster - https://phabricator.wikimedia.org/T75882#784303 (10hashar) [14:20:50] 3Beta-Cluster: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#784336 (10yuvipanda) Statsd runs on labmon1001.eqiad.wmnet for labs. [14:38:07] 3Release-Engineering: Document how to decide if adding a new category collation is OK to do quickly and when we need to do other prep - https://phabricator.wikimedia.org/T447#784406 (10MZMcBride) [14:38:54] 3Release-Engineering: Document how to decide if adding a new category collation is OK to do quickly and when we need to do other prep - https://phabricator.wikimedia.org/T447#4576 (10MZMcBride) [15:58:04] (03PS1) 10Hashar: Generate mw/core doxygen on new release [integration/config] - 10https://gerrit.wikimedia.org/r/175712 [16:00:23] (03CR) 10Hashar: [C: 04-1] "Bah the job has a branch filter:" [integration/config] - 10https://gerrit.wikimedia.org/r/175712 (owner: 10Hashar) [16:06:53] 3Release-Engineering, MediaWiki-Core-Team: Make sure scap rsync proxies aren't in jobrunner pool - https://phabricator.wikimedia.org/T1342#784643 (10bd808) [16:15:25] Project beta-scap-eqiad build #31238: FAILURE in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31238/ [16:17:33] (03PS2) 10Hashar: Generate mw/core doxygen on new release [integration/config] - 10https://gerrit.wikimedia.org/r/175712 [16:28:00] 3Release-Engineering, MediaWiki-Core-Team, Deployment-Systems: Make sure scap rsync proxies aren't in jobrunner pool - https://phabricator.wikimedia.org/T1342#784782 (10greg) [16:29:59] 3Release-Engineering, MediaWiki-Core-Team, Deployment-Systems: Update servers in scap rsync proxy pool - https://phabricator.wikimedia.org/T1342#784784 (10Reedy) [16:31:33] (03CR) 10Hashar: "Seems branch: look at the ref provided by Gerrit ref-updated events according to Zuul source code: https://review.openstack.org/14096" [integration/config] - 10https://gerrit.wikimedia.org/r/175712 (owner: 10Hashar) [16:35:23] Yippee, build fixed! [16:35:24] Project beta-scap-eqiad build #31241: FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31241/ [16:52:38] 3Beta-Cluster: Beta cluster centralauth accounts points to no longer existing wikis - https://phabricator.wikimedia.org/T65396#784831 (10Legoktm) [16:53:28] Project browsertests-Wikidata-WikidataTests-linux-firefox-sauce build #53: FAILURE in 2 hr 31 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-firefox-sauce/53/ [17:00:18] 3Beta-Cluster: Beta cluster centralauth accounts points to no longer existing wikis - https://phabricator.wikimedia.org/T65396#784854 (10Legoktm) > select distinct(lu_wiki) from localuser; 463 rows in set (0.01 sec) > var_dump(count(CentralAuthUser::getWikiList())); int(30) [17:00:21] legoktm: shall I move that to in-progress? ^ [17:00:26] sure [17:01:17] * YuviPanda pokes greg-g with https://phabricator.wikimedia.org/T75866 [17:01:45] greg-g: would be nice if I also got admin on that list, in case the host changes in the future [17:03:03] twentyafterfour: Reedy meeting ping :) [17:03:13] YuviPanda: uno momento por favor, team meeting :) [17:03:24] greg-g: ok! :) [17:04:23] YuviPanda: "Betacluster-alerts post from shinken@shinken-01.eqiad.wmflabs requires approval" things changing again? [17:04:34] greg-g: that's the task I just pointed you to :) [17:04:37] ah, that's the bug you linked ;) [17:04:39] haha [17:07:33] !log ran checkLocalNames.php --delete on all wikis [17:07:38] Logged the message, Master [17:08:39] !log PHP Notice: Undefined index: wmgExtraLanguageNames in /mnt/srv/mediawiki/php-master/includes/SiteConfiguration.php on line 307 [17:08:41] Logged the message, Master [17:09:31] !log ran checkLocalUser.php --delete on all wikis [17:09:33] Logged the message, Master [17:10:32] !log ran migratePass0.php on all wikis [17:10:33] Logged the message, Master [17:12:06] legoktm: That whine has been there for quite a while. I think I filed a bug about it at some point. [17:13:00] legoktm: It's some bug in CommonSettings/InitialiseSettings in beta [17:13:24] The variable is there but not being see at the right time I guess [17:19:15] :/ [17:19:28] I think I'm going to have to write a maint script specifically to fix beta >.> [17:25:35] ugh hangouts totally not cooperating [17:26:06] 3Beta-Cluster: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#784876 (10Reedy) It's probably easiest to just run the script on beta, create it as interwiki-labs.cdb, commit it... ``` if ( file_exists( "$wmfConfigDir/interwiki.cdb" ) ) { $wgInterwikiCache = "$wmfConfigD... [17:28:36] iphone hangouts ftw. linux hangouts: fail [17:32:25] heh [17:36:26] wtffff [17:36:32] http://aa.wikipedia.beta.wmflabs.org/wiki/Main_Page [17:37:16] 3Beta-Cluster: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#784889 (10Reedy) https://gerrit.wikimedia.org/r/175755 [17:45:26] 3Release-Engineering: Don't commit interwiki cdbs - https://phabricator.wikimedia.org/T75905 (10Reedy) 3NEW p:3Triage a:3Reedy [17:48:19] 3MediaWiki-General-or-Unknown, Beta-Cluster: UserMailer silently fails to deliver anything on betacluster - https://phabricator.wikimedia.org/T75736#784915 (10Krenair) 5Open>3Resolved a:3Krenair Works for me now. Thanks Tony. [17:48:34] 3Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded conent - https://phabricator.wikimedia.org/T75206#784918 (10bd808) [17:48:37] 3MediaWiki-General-or-Unknown, Beta-Cluster: UserMailer silently fails to deliver anything on betacluster - https://phabricator.wikimedia.org/T75736#784919 (10Krenair) a:5Krenair>301tonythomas [17:48:49] 3Scrum-of-Scrums, Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded conent - https://phabricator.wikimedia.org/T75206#748930 (10bd808) [17:49:04] 3Scrum-of-Scrums, Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded conent - https://phabricator.wikimedia.org/T75206#748930 (10bd808) [17:53:57] 3MediaWiki-extensions-WikimediaMaintenance, Beta-Cluster: Check/update dumpInterwiki for beta usage - https://phabricator.wikimedia.org/T75906 (10Reedy) 3NEW p:3Normal a:3Reedy [17:55:48] 3Beta-Cluster: Subscribe shinken@shinken-01.eqiad.wmflabs to beta cluster alerts mailing list - https://phabricator.wikimedia.org/T75866#784966 (10greg) [17:57:11] 3Beta-Cluster: Beta cluster centralauth accounts points to no longer existing wikis - https://phabricator.wikimedia.org/T65396#784972 (10greg) [17:57:28] 3Beta-Cluster: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#784973 (10greg) [17:57:55] 3Release-Engineering, Beta-Cluster: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#714124 (10greg) [17:58:14] 3Release-Engineering, Beta-Cluster: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#714124 (10greg) [17:59:32] 3Beta-Cluster: Subscribe shinken@shinken-01.eqiad.wmflabs to beta cluster alerts mailing list - https://phabricator.wikimedia.org/T75866#784981 (10greg) [17:59:37] 3Beta-Cluster: Set up graphite monitoring for the beta cluster - https://phabricator.wikimedia.org/T54357#784982 (10greg) [17:59:42] !log clearing out localnames/localuser where wikis don't exist on beta [17:59:47] Logged the message, Master [17:59:53] 3MediaWiki-General-or-Unknown, Beta-Cluster: UserMailer silently fails to deliver anything on betacluster - https://phabricator.wikimedia.org/T75736#784983 (10greg) [18:00:47] !log 4086 rows deleted from localnames, 3929 from localuser [18:00:49] Logged the message, Master [18:01:51] !log cleared out renameuser_status table (old broken global merges) [18:01:53] Logged the message, Master [18:03:03] 3Beta-Cluster: Beta cluster centralauth accounts points to no longer existing wikis - https://phabricator.wikimedia.org/T65396#784984 (10Legoktm) [18:04:13] greg-g: all done ^ [18:04:43] w00t [18:05:02] 3Beta-Cluster: Beta cluster centralauth accounts points to no longer existing wikis - https://phabricator.wikimedia.org/T65396#784985 (10greg) [18:11:57] 3Release-Engineering, Continuous-Integration: Rename Phabricator tag for "continuous integration" and create a "Zuul" tag - https://phabricator.wikimedia.org/T952#785001 (10greg) [18:17:52] 3Release-Engineering: Can "gem build" fail if there are _any_ warnings? - https://phabricator.wikimedia.org/T1333#785004 (10greg) [18:18:25] 3Release-Engineering, Continuous-Integration: Rename Phabricator tag for "continuous integration" and create a "Zuul" tag - https://phabricator.wikimedia.org/T952#785008 (10greg) 5Open>3Resolved [18:18:32] 3Release-Engineering, Continuous-Integration: Rename Phabricator tag for "continuous integration" and create a "Zuul" tag - https://phabricator.wikimedia.org/T952#16276 (10greg) [18:18:38] 3MediaWiki-Vagrant, Release-Engineering: Analyze MW-Vagrant qualitative survey - https://phabricator.wikimedia.org/T429#785010 (10greg) [18:19:55] 3Release-Engineering: Update scap in prod - https://phabricator.wikimedia.org/T1108#785012 (10greg) [18:22:50] Yippee, build fixed! [18:22:50] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #187: FIXED in 16 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/187/ [18:28:21] 3Release-Engineering, MediaWiki-Core-Team, Deployment-Systems: Update servers in scap rsync proxy pool - https://phabricator.wikimedia.org/T1342#785043 (10greg) p:5Triage>3Normal [18:36:25] 3MediaWiki-General-or-Unknown, Beta-Cluster: UserMailer silently fails to deliver anything on betacluster - https://phabricator.wikimedia.org/T75736#778573 (10greg) (sorry for the noise) [18:42:28] 3Editing, Wikidata, Mobile-Web, Zero, Parsoid-Team, Scrum-of-Scrums, Services, Phabricator, Release-Engineering, Research-and-Data, MediaWiki-Core-Team, Engineering-Community, Language-Engineering, WMF-Design, Multimedia, Core-Features, Mobile-Apps: Create team projects for all teams participating in scrum of scru... [19:02:27] Hmm. Beta Labs is redirecting me to HTTPS… [19:02:30] Known issue? [19:03:21] eh? [19:03:43] James_F: when? any request? [19:03:53] greg-g: It's probably just that some damn-fool script has set the only-HTTPS cookie for me. [19:03:55] greg-g: Yeah. [19:04:00] * greg-g nods [19:04:03] greg-g: But not incognito. [19:04:09] yeah, either script or the dumb safari doesn't do coolies right bug [19:04:14] cookies [19:04:21] Can we please for the love of everything just get a damn SSL cert for Beta Labs? [19:04:30] I'll pay for it if needed. [19:04:31] https://phabricator.wikimedia.org/T72145 [19:04:34] It's just way too disruptive. [19:04:55] greg-g: Not using Safari; this is in Firefox and Chrome. [19:05:25] * greg-g nods [19:06:46] Found it. [19:06:55] what was it? [19:06:58] Set for wikipedia.beta.wmflabs.org [19:07:01] * James_F grumbles. [19:07:13] I need to set a "pissoffSSL" cookie for beta.wmflabs.org [19:07:35] Argh, no. [19:07:43] CentralAuth (?) has re-set it. [19:07:48] ???? [19:07:56] legoktm: ^ :) [19:08:15] uhhhh [19:08:19] Was fine until I reloaded as prompted. [19:08:26] Then the forceHTTPS cookie returned. [19:08:37] hmm, once the EFF's CA takes off I think we could just use those for beta :) [19:09:09] YuviPanda: I'd be happy if we self-signed at this point. [19:09:42] I'm not sure. [19:09:50] I don't think anything I did would be related. [19:09:57] I might have logged people out though. [19:10:06] legoktm: Possible that somehow my user preference to force HTTPS was switched on? [19:10:30] * legoktm checks [19:11:02] James_F: what's your username? [19:13:10] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#785345 (10greg) [19:13:34] legoktm: "Jdforrester (WMF)". [19:13:49] legoktm: I get to "https://login.wikimedia.beta.wmflabs.org/wiki/Special:CentralLogin/start?token=…" and it then fails. [19:14:28] YuviPanda: can we use the same ssl setup that toollabs uses? https://phabricator.wikimedia.org/T56065#590753 [19:14:52] legoktm: Did you do something? It now works. [19:14:58] uhhhh [19:15:00] nope. [19:15:02] :P [19:15:08] legoktm: I'll praise you anyway. :-) [19:15:20] I cleared someone else's cache entry :P [19:15:47] greg-g: sadly not, because that certificate doesn't cover sub-sub domains [19:16:03] legoktm: Ha. [19:16:33] YuviPanda: Can we make ourselves a CA and issue a *.*.*.*.*.*.beta.wmflabs.org cert? [19:16:47] James_F: yes, if we paid another CA a *lot* of money :) [19:16:49] YuviPanda: I'd accept the cert locally. [19:16:55] that [19:16:58] you can do, yes. [19:17:02] it was the case initially [19:17:04] YuviPanda: We only need it for testers of Beta Labs. [19:17:06] and was turned off, I think. [19:17:10] We can expect us to install a cert. [19:17:12] It's not hard. [19:17:14] because it was causing issues with browsertests? [19:17:18] I'm not sure. [19:17:22] Then we need to fix browser tests? [19:17:27] https://gerrit.wikimedia.org/r/#/c/87045/ [19:17:50] https://phabricator.wikimedia.org/T50501 has gory details [19:18:19] * James_F sighs. [19:19:12] James_F: just accepting warning won't work, because there will be no warning for bits [19:19:27] YuviPanda: I can install the cert locally, though. [19:19:32] yup [19:19:36] YuviPanda: Takes a few seconds. [19:19:41] yup [19:19:43] And avoids wasted hours on this nonsense. [19:20:01] yup as well [19:20:11] the problem is discoverability [19:20:51] that'll work for browser tests/people who know, but when someone tries to test something for the first time (say, a recruited beta tester/whatever) they'll get a warning/error and not know what to do, we can't put an interstitial there [19:21:22] (where "that" == "just install the self-signed cert locally") [19:21:46] I dunno if that's worse than status quo [19:22:04] well, it locks people out [19:22:09] now they just sometimes hit the issue [19:22:18] (I haven't hit it in a looonnng time, but I don't do extensive testing) [19:22:33] * James_F grumbles. [19:23:05] bc if we did "just like prod" it's https by default, which means by default you're locked out/getting that error (which we can't explain) [19:23:58] https://letsencrypt.org/ should help fix that, I think [19:24:00] at worst [19:24:00] If the quoted number I have from before is what it would cost, I'm willing to lobby for it (and could probably get it), iff we can deal with the management aspect of the private certs [19:24:39] YuviPanda: do you remember the bug describing why we can't/it's hard to manage the private cert? [19:25:03] greg-g: same as https://phabricator.wikimedia.org/T50501#527366 but we have that fixed now with the NDA group [19:25:06] on betalabs [19:25:34] https://phabricator.wikimedia.org/T71269#719261 is also relevant [19:26:26] yeah, that too [19:26:50] you need to have the NDA group if we want real certs [19:26:54] but we already have nda group now [19:26:55] but no real certs [19:27:03] so kind of 'disadvantages of both worlds' here [19:27:19] well, because sudo on beta cluster isn't limited to those with an NDA anymore right? [19:27:41] it isn't? [19:27:44] I thought it still was/ [19:27:44] ? [19:27:49] oh, it's just proposed [19:27:58] yeah, that never happened [19:28:03] mostly because we hired everyone, I think [19:28:04] >_> [19:28:14] :) [19:28:22] and John Lewis is nda covered now [19:28:24] yeah [19:28:43] and also have more roots around, so people who aren't root aren't as handicapped [19:29:34] greg-g: so management wouldn't be too much of an issue, I think [19:33:02] YuviPanda: so, we could, with the current situation (I won't hold you to this, just getting your understanding) buy the certs listed at https://phabricator.wikimedia.org/T50501, place them on the machines limited by sudo, and be ok? or is there more there I'm missing? (ie: is labs inherently unsafe?) [19:33:39] YuviPanda: also/and/or/maybe, can we do a similar thing you did for toollabs, but specific to beta cluster? I assume that is a machine between the web and toollabs that holds the certs and thus is safe [19:33:56] yup. that won't mirror prod, but that would be the safest option. [19:34:00] (sorry to bogart you time right now on this) [19:34:03] * greg-g nods [19:34:12] that might be good enough? [19:34:24] I'll try to summarize somewhere [19:34:27] tch tch, I was hired for fixing labs issues :) [19:34:42] yeah, I think that would be good enough. [19:34:52] it might also mean we can actually allow everyone to have full root in deployment-prep itself [19:34:57] and just restrict on the terminator [19:35:04] yeah [19:35:06] and it would be a fairly simple terminator too. [19:35:11] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #317: STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/317/ [19:35:15] so we'll get 'everyone has sudo' and 'real certs', just not 'exact same as prod' [19:35:26] so, I'll start a new bug (yeah yeah, I think it'll keep things clearer) and summarize this conversation/proposal [19:35:32] yeah. [19:35:36] cool, thanks sirt [19:35:36] 1. buy real certs [19:35:37] -t [19:35:44] 2. make new project, restrict access a lot [19:35:55] 3. just ssl terminate there, proxy back to regular deployment-prep setup [19:36:03] greg-g: :) do cc me. [19:37:20] * greg-g nods [19:39:43] greg-g: also note that we can do this with a self-signed cert easily to start with, and then buy the real certs later. or also just wait for the EFF :) [19:39:54] with lets-encrypt the monetary cost will be 0 [19:41:00] I'm not going to wait on vapor ware :) [19:41:07] haha :) [19:41:15] well, you've more experience with them, so I'll take your word for it :) [19:41:20] * YuviPanda has high hopes for it [19:50:12] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #306: SUCCESS in 57 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/306/ [19:57:36] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785524 (10greg) [19:57:56] YuviPanda: ^ :) [19:59:08] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #377: FAILURE in 1 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/377/ [20:02:58] 3MediaWiki-extensions-GettingStarted, Beta-Cluster: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#785567 (10greg) [20:03:00] 3Beta-Cluster: Add RecentActivityFeed extension to beta labs - https://phabricator.wikimedia.org/T71785#785570 (10greg) [20:03:03] 3CirrusSearch, Beta-Cluster: Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#785571 (10greg) [20:03:04] 3Beta-Cluster: beta labs no longer listens for HTTPS - https://phabricator.wikimedia.org/T70387#785572 (10greg) [20:03:07] 3VisualEditor, Beta-Cluster: Beta Cluster: API PrefixSearch is taking a very long time to return, and returns nothing when it does - https://phabricator.wikimedia.org/T74332#785573 (10greg) [20:03:11] 3MediaWiki-User-login-and-signup, Beta-Cluster: Safari sets forceHTTPS=deleted incorrectly, causing login failure on Beta Cluster - https://phabricator.wikimedia.org/T72145#785576 (10greg) [20:03:13] 3Beta-Cluster, Mobile-Web: Can't edit mobile site on beta labs - https://phabricator.wikimedia.org/T75770#785577 (10greg) [20:03:17] 3Beta-Cluster: Rename all occurences of "deployment-prep" to "beta-cluster" - https://phabricator.wikimedia.org/T74694#785579 (10greg) [20:03:18] 3Beta-Cluster: API error: unable to connect to redis - https://phabricator.wikimedia.org/T74982#785582 (10greg) [20:18:23] 3MediaWiki-extensions-GettingStarted, Beta-Cluster: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#785709 (10greg) p:5Low>3Triage I can't repro this anymore, when I append ?tour=gettingstarted to any page (on beta cluster) I don't get any popup at all (?!). [20:18:33] 3MediaWiki-extensions-GettingStarted, Beta-Cluster: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#785711 (10greg) [20:18:38] 3Beta-Cluster: API error: unable to connect to redis - https://phabricator.wikimedia.org/T74982#785712 (10greg) [20:18:42] 3Beta-Cluster: Rename all occurences of "deployment-prep" to "beta-cluster" - https://phabricator.wikimedia.org/T74694#785713 (10greg) [20:18:44] 3Beta-Cluster, Mobile-Web: Can't edit mobile site on beta labs - https://phabricator.wikimedia.org/T75770#785714 (10greg) [20:18:51] 3MediaWiki-User-login-and-signup, Beta-Cluster: Safari sets forceHTTPS=deleted incorrectly, causing login failure on Beta Cluster - https://phabricator.wikimedia.org/T72145#785715 (10greg) [20:18:55] 3VisualEditor, Beta-Cluster: Beta Cluster: API PrefixSearch is taking a very long time to return, and returns nothing when it does - https://phabricator.wikimedia.org/T74332#785716 (10greg) [20:18:57] 3Beta-Cluster: beta labs no longer listens for HTTPS - https://phabricator.wikimedia.org/T70387#785717 (10greg) [20:19:00] 3CirrusSearch, Beta-Cluster: Search is sometimes slow on the Beta Cluster - https://phabricator.wikimedia.org/T72869#785718 (10greg) [20:19:04] 3Beta-Cluster: Add RecentActivityFeed extension to beta labs - https://phabricator.wikimedia.org/T71785#785719 (10greg) [20:19:46] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785722 (10greg) p:5Triage>3Normal [20:19:53] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785524 (10greg) [20:24:24] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#785728 (10greg) I was trying to repro and hit 503s from upload as well, just loading the main page (the main logo didn't... [20:24:39] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#785732 (10greg) [20:26:20] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#785745 (10greg) [20:26:26] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#785749 (10greg) [20:28:22] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#785763 (10greg) [20:28:35] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#785763 (10greg) [20:28:38] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#785776 (10greg) [20:28:41] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#785777 (10greg) [20:28:52] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#785780 (10greg) p:5High>3Normal [20:29:02] 3Beta-Cluster: mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - https://phabricator.wikimedia.org/T67591#785783 (10greg) [20:30:46] 3Beta-Cluster: beta cluster: deployment-cache-upload02 does not seem to purge content when getting PURGE - https://phabricator.wikimedia.org/T67683#785784 (10greg) [20:33:30] 3Scrum-of-Scrums, Beta-Cluster: beta cluster: deployment-cache-upload02 does not seem to purge content when getting PURGE - https://phabricator.wikimedia.org/T67683#785788 (10greg) [20:33:40] 3Scrum-of-Scrums, Beta-Cluster: beta cluster: deployment-cache-upload02 does not seem to purge content when getting PURGE - https://phabricator.wikimedia.org/T67683#698061 (10greg) [20:33:55] 3Beta-Cluster: sync articles from production wikis (css/gadgets) - https://phabricator.wikimedia.org/T51779#785791 (10greg) [20:33:59] 3Beta-Cluster: sync Sandbox gadget from production to en.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T49205#785792 (10greg) [20:34:03] 3Beta-Cluster: automatically import some content from production (tracking) - https://phabricator.wikimedia.org/T54382#785794 (10greg) [20:34:05] 3Beta-Cluster: sync-site-resources should sync all Labs wikis - https://phabricator.wikimedia.org/T51791#785796 (10greg) [20:34:10] 3Beta-Cluster: broken Upload file link in en-beta navigation - https://phabricator.wikimedia.org/T59117#785798 (10greg) [20:37:15] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785800 (10bd808) > make new labs project, restrict access a lot The only downside I see for this plan is that it becomes "yet another way that beta is not lik... [20:42:04] 3Release-Engineering, Scrum-of-Scrums, MediaWiki-Core-Team: disks for logstash - https://phabricator.wikimedia.org/T1225#785813 (10bd808) [20:42:07] 3Editing, Wikidata, Mobile-Web, Zero, Parsoid-Team, Scrum-of-Scrums, Services, Phabricator, Release-Engineering, Research-and-Data, MediaWiki-Core-Team, Engineering-Community, Language-Engineering, WMF-Design, Multimedia, Core-Features, Mobile-Apps: Create team projects for all teams participating in scrum of scru... [20:46:39] 3MediaWiki-extensions-GettingStarted, Beta-Cluster: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#785824 (10hashar) [20:52:55] 3Release-Engineering, Beta-Cluster: Use Beta cluster as a true canary for code deployments (tracking) - https://phabricator.wikimedia.org/T53494#785863 (10greg) [20:52:58] 3Release-Engineering, Beta-Cluster: Convert Beta Cluster specific puppet configs to use Hiera (tracking) - https://phabricator.wikimedia.org/T451#785865 (10greg) [20:53:05] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#785867 (10yuvipanda) Can't actually use dynamicproxy, since the cert there is just for *.wmflabs.org. Also, this terminator could be much simpler, since all it... [20:57:02] 3Beta-Cluster: Unable to upload new version of images in commons beta lab - https://phabricator.wikimedia.org/T70760#785883 (10greg) 5Open>3Resolved a:3greg Resolving per: >>! In T70760#719880, @bd808 wrote: > Beta uses the NFS shared directory /data/project/upload7 to store images. > > I just did a per... [20:57:19] 3Beta-Cluster: Unable to upload new version of images in commons beta lab - https://phabricator.wikimedia.org/T70760#785888 (10greg) [20:59:47] 3Scrum-of-Scrums, Beta-Cluster: beta cluster: deployment-cache-upload02 does not seem to purge content when getting PURGE - https://phabricator.wikimedia.org/T67683#785892 (10JeanFred) [21:00:56] 3MediaWiki-extensions-GettingStarted, Beta-Cluster: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#785898 (10greg) [21:04:11] 3Scrum-of-Scrums, Beta-Cluster: File upload area resorts to 0777 permissions to for uploaded content - https://phabricator.wikimedia.org/T75206#785918 (10hashar) [21:04:51] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #151: FAILURE in 1 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/151/ [21:05:40] !log Running `sudo find . -type d ! -perm -o=w -exec chmod 0777 {} +` to fix upload permissions [21:05:42] Logged the message, Master [21:13:57] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#786000 (10Qgil) [21:20:14] Yippee, build fixed! [21:20:14] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #324: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/324/ [21:24:18] Hey I can't load images on beta labs - Failed to load resource: the server responded with a status of 503 (Service Unavailable) [21:24:58] Also VE isn't loading on beta labs - TypeError: Cannot read property 'previousSibling' of null TypeError: Cannot read property 'previousSibling' of null {stack: (...), message: "Cannot read property 'previousSibling' of null"} [21:25:39] kaity: (hi) re: the VE error, the first JS error I see is Exception thrown by papaparse" load.php:161 [21:25:42] "TypeError: document.getElementById(...) is null" TypeError: document.getElementById(...) is null [21:25:50] try mama parse instead :) [21:26:10] FWIW I'm seeing the images OK on http://en.wikipedia.beta.wmflabs.org/wiki/Dido_Sotiriou?veaction=edit [21:26:29] James_F: ^ error with VE on beta labs [21:27:25] ryasmeen: ^ [21:27:34] kaity: spagewmf it's known, I reported it early this morning [21:28:24] spagewmf: The image thing is also reported in a separate bug yesterday [21:30:55] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#786050 (10greg) [21:31:36] 3MediaWiki-extensions-WikimediaMaintenance, Beta-Cluster: Check/update dumpInterwiki for beta usage - https://phabricator.wikimedia.org/T75906#786056 (10greg) [21:35:52] 3Beta-Cluster: [OPS] udp2log prevents udp2log-mw from starting - https://phabricator.wikimedia.org/T40995#786062 (10greg) [21:38:04] 3Beta-Cluster: Caching makes it impossible to test JS changes when logged out - https://phabricator.wikimedia.org/T65034#786070 (10greg) [21:39:22] 3Deployment-Systems, Beta-Cluster: [trebuchet] salt commands exceed timeout - https://phabricator.wikimedia.org/T72863#786081 (10greg) Antoine: is this still happening? [21:39:33] 3Deployment-Systems, Beta-Cluster: [trebuchet] salt commands exceed timeout - https://phabricator.wikimedia.org/T72863#786083 (10greg) [21:50:16] 3MediaWiki-extensions-CentralAuth, Beta-Cluster: CentralAuth logging a lot (login attempts/successes/etc) on Beta Cluster - https://phabricator.wikimedia.org/T75929#786097 (10greg) [21:51:33] 3MediaWiki-JobRunner, Beta-Cluster: beta cluster job runner keep running some periodic tasks - https://phabricator.wikimedia.org/T65681#786108 (10greg) >>! In T65681#710874, @bzimport wrote: > **aschulz4587** wrote: > > If the add() gave NOT STORED then you should have seen cas(). That sounds like some problem... [21:51:46] 3MediaWiki-JobRunner, Beta-Cluster: beta cluster job runner keep running some periodic tasks - https://phabricator.wikimedia.org/T65681#786111 (10greg) p:5Normal>3Triage [21:53:28] 3Beta-Cluster: setup a DB backed parser cache - https://phabricator.wikimedia.org/T55457#786118 (10greg) @springle: Guidance here? [21:53:37] 3Beta-Cluster: setup a DB backed parser cache - https://phabricator.wikimedia.org/T55457#786121 (10greg) [21:55:05] 3MediaWiki-Maintenance-scripts, Beta-Cluster: Interlanguage links do not work inside labs environment - https://phabricator.wikimedia.org/T50299#786123 (10greg) [21:55:25] 3MediaWiki-Maintenance-scripts, Beta-Cluster: Interlanguage links do not work inside labs environment - https://phabricator.wikimedia.org/T50299#786127 (10greg) p:5Normal>3Volunteer? [21:56:04] 3Beta-Cluster: Reenable $wgMWOAuthSecureTokenTransfer=true; - https://phabricator.wikimedia.org/T67421#786129 (10greg) [22:00:08] 3MediaWiki-Core-Team, MediaWiki-JobRunner, Beta-Cluster: beta cluster job runner keep running some periodic tasks - https://phabricator.wikimedia.org/T65681#786131 (10hashar) [22:01:06] 3MediaWiki-Core-Team, MediaWiki-JobRunner, Beta-Cluster: beta cluster job runner keep running some periodic tasks - https://phabricator.wikimedia.org/T65681#786133 (10hashar) [22:02:53] 3Release-Engineering, Scrum-of-Scrums, MediaWiki-Core-Team: Performance review BounceHandler extension for Deployment - https://phabricator.wikimedia.org/T1227#786136 (10bd808) [22:08:51] 3MediaWiki-extensions-CentralAuth, Beta-Cluster: CentralAuth logging a lot (login attempts/successes/etc) on Beta Cluster - https://phabricator.wikimedia.org/T75929#786170 (10Legoktm) All of those are logged in prod as well. The migration stuff has been useful in debugging weird errors imo. [22:13:18] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #221: FAILURE in 2 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/221/ [22:26:33] 3MediaWiki-Core-Team, MediaWiki-JobRunner, Beta-Cluster: beta cluster job runner keep running some periodic tasks - https://phabricator.wikimedia.org/T65681#786226 (10hashar) [22:29:28] (03PS1) 10Dduvall: Ignore additional RuboCop Metric/* rules [ruby/api] - 10https://gerrit.wikimedia.org/r/175870 [22:31:06] (03CR) 10Dduvall: "Looks like I forgot to add reviewers. :)" [ruby/api] - 10https://gerrit.wikimedia.org/r/170189 (https://bugzilla.wikimedia.org/63307) (owner: 10Dduvall) [22:31:50] (03PS2) 10Dduvall: Ignore additional RuboCop Metric/* rules [ruby/api] - 10https://gerrit.wikimedia.org/r/175870 (https://bugzilla.wikimedia.org/175870) [22:34:22] ryasmeen: if it is of interest, marxarelli is the one who can tell you about the difference between class variables, global variables, and constants in Ruby. (I know we abuse all of them in various places) [22:35:32] Project beta-scap-eqiad build #31278: FAILURE in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31278/ [22:36:00] indeed! [22:37:36] ah thanks chrismcmahon! yeah marxarelli, maybe we will chat about it next time I see you [22:37:39] :) [22:37:43] marxarelli: ryasmeen asked some good questions about the difference between global variables and class variables, and I was quickly out of my depth [22:39:57] chrismcmahon, ryasmeen: the short story about both of them is: it's a trap! :) [22:40:22] haha! [22:41:05] unfortunately, class variables are implemented in a pretty terrible way in Ruby [22:41:07] 3MediaWiki-extensions-GettingStarted, Beta-Cluster: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#786283 (10greg) [22:41:38] e.g. with class A; @@some_var = 'x'; end [22:41:46] and class B; @@some_var = 'y'; end [22:41:53] marxarelli: wait, what's the @foo var called? [22:42:04] single-@ not double-@@ [22:42:13] A's @@some_var will equal 'y'! [22:42:39] i.e. their state is shared among their entire class hierarchy [22:42:43] I always have to read the docs whenever I start messing with @foo @@bar $baz [22:42:44] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786288 (10greg) [22:42:52] chrismcmahon: oh, that's an instance variable [22:43:03] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#785763 (10greg) [22:43:09] ryasmeen, chrismcmahon: those are alright :) [22:43:44] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#786291 (10greg) [22:44:05] ryasmeen, chrismcmahon: short story there: they only have visibility from within the class [22:44:17] so the @@some_var within class A gets overridden by Class B? [22:44:27] ryasmeen: yes, exactly [22:44:36] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#786292 (10hashar) I loose track on the rather long {T50501} , but I commented about using a certificate authority to generate free certs for beta then inject... [22:44:43] so, never use them... [22:44:47] hmmm.. [22:45:06] ryasmeen: https://github.com/bbatsov/ruby-style-guide#no-class-vars [22:45:59] so thats defined as rules in our rubocop? [22:46:07] ryasmeen: but instance variables are very useful to store state within the scope of a class instance [22:46:14] ryasmeen: right [22:47:22] i see! [22:48:15] in ruby, there's a very basic pattern that everything is built upon: obj.send(:method, *arguments) [22:48:15] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Setup real ssl certs for Beta Cluster using a restricted project - https://phabricator.wikimedia.org/T75919#786302 (10greg) >>! In T75919#786292, @hashar wrote: > I loose track on the rather long {T50501} , but I commented about using a certificate authority to gener... [22:48:26] almost everything can be reduced to that [22:49:26] i.e. sending messages to objects and getting back a value [22:49:45] anyway, i'm blathering... [22:49:57] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#786303 (10greg) [22:50:15] 3Deployment-Systems, Beta-Cluster: [trebuchet] salt commands exceed timeout - https://phabricator.wikimedia.org/T72863#786304 (10hashar) [22:51:02] oh no, I have got it so far :D [22:51:02] 3Deployment-Systems, Beta-Cluster: [trebuchet] salt commands exceed timeout - https://phabricator.wikimedia.org/T72863#786314 (10hashar) [22:51:09] 3MediaWiki-extensions-BounceHandler, Beta-Cluster: Beta config needs update (Undefined variable: wmgVERPsecret) - https://phabricator.wikimedia.org/T75757#786315 (10hashar) [22:51:21] so in the browser tests we never deal with multiple classes, so we never need scope beyond instance vars e.g. @wikitext EXCEPT when we always need the current browser $browser which must be global [22:52:30] ryasmeen: cool! so, how that relates to instance variables is that you can't access them directly from outside an object's instance scope. you have to expose them with methods [22:53:03] ryasmeen: e.g. class A; @foo; def foo; @foo; end; end [22:53:26] ryasmeen: allows you to access the state of @foo by way of: obj.foo [22:53:54] okay from a different class or method [22:54:32] got it [22:54:47] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#786337 (10Jdforrester-WMF) [22:54:51] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#786338 (10Jdforrester-WMF) [22:55:06] ryasmeen: great! [22:55:32] Yippee, build fixed! [22:55:33] Project beta-scap-eqiad build #31280: FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/31280/ [22:56:14] 3Beta-Cluster, Deployment-Systems: rsync errors to beta cluster, inconsistent state after scap - https://phabricator.wikimedia.org/T71590#786351 (10greg) [22:56:22] 3Beta-Cluster, Deployment-Systems: rsync errors to beta cluster, inconsistent state after scap - https://phabricator.wikimedia.org/T71590#725948 (10greg) [23:13:35] 3Beta-Cluster, Deployment-Systems: rsync errors to beta cluster, inconsistent state after scap - https://phabricator.wikimedia.org/T71590#786434 (10greg) >>! In T71590#726083, @hashar wrote: > All good. > > > Lowering priority of the bug since it is hacked/manually fixed. I am leaving it open until the Gerrit... [23:15:10] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #387: FAILURE in 1 hr 23 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/387/ [23:17:07] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#786448 (10Jdforrester-WMF) 5Open>3Resolved [23:17:25] 3Beta-Cluster, VisualEditor: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#786450 (10Jdforrester-WMF) 5Open>3Resolved [23:18:12] 3Beta-Cluster, Deployment-Systems: rsync errors to beta cluster, inconsistent state after scap - https://phabricator.wikimedia.org/T71590#786458 (10hashar) 5Open>3Resolved The patch I created ( https://gerrit.wikimedia.org/r/#/c/154329/ ) was a revert which I cherry picked on the beta cluster to immediately... [23:21:21] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786477 (10greg) The two VE-related bugs are closed, but I'm still getting a 503 when trying to load the Beta Cluster logo: http://upload.beta.wmflabs.org/wikipedia/en/b/bc/Wiki.png [23:21:50] 3Beta-Cluster, Deployment-Systems: rsync errors to beta cluster, inconsistent state after scap - https://phabricator.wikimedia.org/T71590#786478 (10greg) [23:21:57] 3Beta-Cluster, Deployment-Systems: rsync errors to beta cluster, inconsistent state after scap - https://phabricator.wikimedia.org/T71590#725948 (10greg) [23:22:04] 3Beta-Cluster, MediaWiki-Maintenance-scripts: Interlanguage links do not work inside labs environment - https://phabricator.wikimedia.org/T50299#500339 (10hashar) [23:22:26] James_F: just fyi: https://phabricator.wikimedia.org/T75922#786477 [23:22:44] 3Beta-Cluster, Release-Engineering: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#786485 (10hashar) 5duplicate>3Open [23:22:56] 3Beta-Cluster, MediaWiki-Maintenance-scripts: Interlanguage links do not work inside labs environment - https://phabricator.wikimedia.org/T50299#786486 (10hashar) [23:22:58] 3Beta-Cluster, Release-Engineering: Beta should not use productions interwiki.cdb - https://phabricator.wikimedia.org/T69931#714124 (10hashar) [23:23:03] greg-g: Yeah, I think either it magically fixed and then broke itself or else I just got lucky with caches. [23:23:58] 3VisualEditor, Beta-Cluster: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#786490 (10Jdforrester-WMF) [23:23:58] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786492 (10Jdforrester-WMF) [23:23:59] 3VisualEditor, Beta-Cluster: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#786491 (10Jdforrester-WMF) [23:24:01] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#785763 (10Jdforrester-WMF) [23:24:01] ryasmeen, greg-g: Have merged the VE bugs into this. [23:24:02] 3Beta-Cluster, MediaWiki-Maintenance-scripts: Interlanguage links do not work inside labs environment - https://phabricator.wikimedia.org/T50299#786499 (10greg) [23:24:24] 3Mobile-Web, Beta-Cluster: Can't edit mobile site on beta labs - https://phabricator.wikimedia.org/T75770#786505 (10Etonkovidova) There were some problems with betalabs env. Checked http://en.m.wikipedia.beta.wmflabs.org on iPhone - Edit is functioning. [23:25:18] 3Mobile-Web, Beta-Cluster: Can't edit mobile site on beta labs - https://phabricator.wikimedia.org/T75770#786509 (10Etonkovidova) [23:26:25] 3Mobile-Web, Beta-Cluster: Can't edit mobile site on beta labs - https://phabricator.wikimedia.org/T75770#786514 (10greg) p:5Triage>3Volunteer? [23:28:52] greg-g: This is one of those unbreak-now level issues, almost. It's pretty hard for QA to test VE the day before the Thanksgiving deploy… :-( [23:30:07] James_F: +1 [23:30:56] 3Beta-Cluster, MediaWiki-extensions-GettingStarted: GettingStarted extension broken at beta.wmflabs.org - https://phabricator.wikimedia.org/T53362#786537 (10hashar) I have added @Mattflaschen @rmoen and @phuedx as members of the #MediaWiki-extensions-GettingStarted project. [23:31:08] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786538 (10greg) [23:32:04] James_F: ryasmeen I almost reported it as 'unbreak now' this morning, but y'all usually recover from errors like this greg-g [23:32:16] chrismcmahon: Yeah… [23:32:40] chrismcmahon: Thing is, "Unbreak now!" puts it on the dashboard of every single user of Phabricator Wikimedia wide, which is… rude. [23:32:44] I reported it as "high" (or whatever) instead, I think it had about the same effect [23:34:42] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786546 (10hashar) a:3hashar Sure thing. [23:35:06] Yippee, build fixed! [23:35:07] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #311: FIXED in 1 hr 21 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/311/ [23:35:20] chrismcmahon: In the default view you only see "High" bugs for your projects, but you see "Unbreak now!" ones for all projects. [23:35:27] chrismcmahon: It's a pretty nuclear option. :-) [23:35:30] yep [23:35:47] didn't seem nuclear at 7AM PST [23:36:02] (03PS3) 10Dduvall: Ignore additional RuboCop Metric/* rules [ruby/api] - 10https://gerrit.wikimedia.org/r/175870 (https://bugzilla.wikimedia.org/175870) [23:40:02] James_F: where do you see that? [23:40:25] (03CR) 10Dduvall: [C: 04-1] "Incidentally, I've submitted a patch that ignores this RuboCop rule, but I like your refactoring nonetheless." (031 comment) [ruby/api] - 10https://gerrit.wikimedia.org/r/175686 (owner: 10Stan) [23:40:28] greg-g: Depends on your Maniphest view, perhaps? [23:40:45] at https://phabricator.wikimedia.org/maniphest/ ? [23:41:07] Certainly it appears there. [23:41:32] But only because it's the "open tasks" view for anons. [23:41:43] For logged-in users it defaults to "assigned", and it doesn't show there. [23:41:44] * James_F shrugs. [23:41:53] Maybe I should be more liberal in using it, then. [23:44:24] 3VisualEditor, Beta-Cluster: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so all images are appearing with a broken icon inside VE - https://phabricator.wikimedia.org/T75786#786599 (10hashar) [23:44:26] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786596 (10hashar) [23:44:27] 3VisualEditor, Beta-Cluster: [Regression pre-wmf10] upload.beta.wmflabs.org is throwing 503s so Math function parsing is completely broken inside VE - https://phabricator.wikimedia.org/T75787#786600 (10hashar) [23:45:39] !log Fixed upload cache on beta cluster, the Varnish backend had a mmap SILO error that prevented the backend from starting. https://phabricator.wikimedia.org/T75922 [23:45:40] Logged the message, Master [23:45:43] greg-g: ^^^ :-] [23:46:11] hashar: awesome, thanks! [23:46:44] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786609 (10hashar) For later reference: ``` deployment-cache-upload02:~$ apt-cache policy varnish varnish: Installed: 3.0.5plus~x-wm7 Candidate: 3.0.5plus~x-wm7 Version table: *** 3.0.5plus~x-wm7 0 10... [23:46:57] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#786610 (10Ryasmeen) Thanks for unbreaking this one.All related bugs are resolved now. [23:47:00] hashar: can you file that follow task for brandon? it's way out of my league :) [23:47:27] hashar: also, thanks a ton again for the late night save :) [23:47:31] now get off irc ;) [23:51:39] (03PS2) 10Dduvall: module/class doc comments (per rubocop) [ruby/api] - 10https://gerrit.wikimedia.org/r/175687 (owner: 10Stan) [23:52:31] 3Beta-Cluster: upload.beta.wmflabs.org is throwing 503s - https://phabricator.wikimedia.org/T75922#785763 (10hashar) I have filled the stacktrace as a task for ops to investigate: {T75943} [23:53:12] greg-g: I find it annoying to close the task then have to go to the board, show closed task, find the task and move it from the bottom left corner (backlog column) up to the top right corner (done column) [23:53:39] greg-g: maybe we should prevent task from being closed :-] [23:53:59] thus folks would move it to done and it is up to the project lead/manager to close them from the done column [23:54:02] moaar process [23:54:52] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #296: FAILURE in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/296/ [23:54:53] hashar: yeah, one method that James/Editing use is basically: only he or Rummana/Elena can actually close the task, devs just move it to Done [23:55:01] chrismcmahon: upload.beta.wmflabs.org was throwing 503 due to varnish being dead. Fixed it a few minutes ago ( https://phabricator.wikimedia.org/T75922 ) [23:55:14] hashar: got it, thank you! [23:55:32] greg-g: cause obviously devs have no idea what they are doing. That seems fine to me and I think that is what scrum propose [23:55:41] :) :)