[00:51:22] 10Beta-Cluster, 10VisualEditor, 10VisualEditor-MediaWiki: On Beta Cluster, switching from VisualEditor to edit source mode intermittently loads the wikitext editor without any CSS - https://phabricator.wikimedia.org/T86624#1111775 (10Krenair) a:5Krenair>3None [02:12:17] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<12.50%) [02:35:49] 10Deployment-Systems: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1111915 (10bd808) scap and sync-* make the ssh connection as the mwdeply user. The l10nupdate ssh key is only good for itself and not this shared deplo... [02:47:24] 10Deployment-Systems, 5Patch-For-Review: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1111938 (10bd808) >>! In T76061#1111915, @bd808 wrote: > scap and sync-* make the ssh connection as the mwdeply user. The l10nupda... [03:14:36] 10Staging, 5Patch-For-Review: Create staging-db* (databases) - https://phabricator.wikimedia.org/T91545#1112018 (10Springle) >>! In T91545#1109576, @yuvipanda wrote: > @springle How would this be handled for master, where there's nowhere to clone from? Masters and slaves are the same. Cloning a new master is... [03:23:47] 10Staging, 5Patch-For-Review: Create staging-db* (databases) - https://phabricator.wikimedia.org/T91545#1112023 (10Springle) >>! In T91545#1109522, @thcipriani wrote: > Next steps: make manual steps ↑ as painless as possible. It would be nice to make de/re-pooling not require a medawiki-config commit. That's... [03:29:27] 10Continuous-Integration, 7Jenkins: PHP fatal errors are not visible in jenkins output in Wikibase phpunit jobs - https://phabricator.wikimedia.org/T92397#1112036 (10Legoktm) It appears to be limited to the mwext-Wikibase-client-tests job, the repo ones are fine. [03:45:58] Yippee, build fixed! [03:45:58] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #365: FIXED in 38 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/365/ [03:46:05] greg-g: triggered a bunch of fatals ("Class 'GlobalUsage' not found") by running a maintenance script on the wrong wikis [03:46:13] they are harmless, sorry for the noise [03:56:28] 10Continuous-Integration, 10MediaWiki-Codesniffer, 10Possible-Tech-Projects, 3Google-Summer-of-Code-2015, 3Outreachy-Round-10: Improving static analysis tools for MediaWiki - https://phabricator.wikimedia.org/T89682#1112054 (10devunt) I'm interested at this project and I'm willing to apply GSoC as a cand... [04:05:28] tgr: :) thanks [05:01:57] I may have actually fixed l10nupdate this time [05:02:07] And i'm sort of stoked about that [05:02:57] The last piece was to tell sync-dir to connect to the remote hosts as the l10nupdate user instead of as the mwdeploy user [05:03:03] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 71.43% of data above the critical threshold [0.0] [05:06:45] bd808: !!! [05:07:21] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:08:00] YuviPanda: is that you ^ (the puppet failures)? [05:09:16] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [05:09:21] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #534: FAILURE in 35 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/534/ [05:09:35] yuvipanda: are those puppet failures you? [05:09:43] yup yup [05:09:46] transient [05:09:46] kk [05:09:48] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [05:09:55] wmf-insecte killed itself [05:10:00] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [05:10:04] oh. [05:10:10] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [05:10:47] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [05:11:29] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [0.0] [05:11:41] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [05:11:45] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [05:11:49] greg-g: basically, two patches were merged, and puppet hit these in the time inbetween (I had to rebase again because someone else merged something in the interim) [05:11:51] is all good now [05:12:50] Guest56166: gotcha [05:15:48] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [05:16:10] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 85.71% of data above the critical threshold [0.0] [05:21:29] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:24:44] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:25:32] greg-g: betalabs scap is going to break for a few mins as well [05:26:18] *cough* beta cluster [05:32:25] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [05:34:23] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [05:35:23] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:36:35] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [05:38:07] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:40:10] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [05:41:10] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:06:45] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:10:43] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [06:17:32] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:17:34] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [06:19:04] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [06:19:04] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0] [06:20:12] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [06:21:35] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [06:22:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [06:25:09] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0] [06:25:15] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [06:25:15] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [06:25:45] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:25:59] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:26:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [06:27:59] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:28:09] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:30:18] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [06:31:12] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [06:31:18] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 85.71% of data above the critical threshold [0.0] [06:31:24] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [06:31:26] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [06:32:58] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [06:33:06] PROBLEM - Puppet failure on deployment-parsoid01-test is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [0.0] [06:33:24] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:33:50] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:35:46] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [06:37:14] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [07:07:24] PROBLEM - Puppet staleness on deployment-test is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0] [07:11:23] !log rebooted puppetmaster, was dead [07:11:29] Logged the message, Master [07:11:51] !log scap still failing on beta, I'll check when I'm back from lunch [07:11:55] Logged the message, Master [07:19:05] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:13] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:15] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:47] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:50] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:23:04] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [07:26:20] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:26:20] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [07:26:36] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:26:46] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [07:26:48] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [07:27:56] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:28:05] RECOVERY - Puppet failure on deployment-parsoid01-test is OK: OK: Less than 1.00% above the threshold [0.0] [07:28:49] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [07:30:17] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:31:28] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:33:31] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:15] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:08] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:36:18] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [07:37:36] RECOVERY - Puppet failure on deployment-sentry2 is OK: OK: Less than 1.00% above the threshold [0.0] [07:37:36] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:08] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:40:16] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:40:50] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:40:50] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:25:04] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [08:40:08] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:42:04] This is Manoj. Working in Bangalore,India as a Principal QA Engineer for one of the SF based ecommerce company [08:56:35] 10Continuous-Integration, 10Flow: Jenkins reports test failures in current master: Cannot override frozen service "storage" - https://phabricator.wikimedia.org/T91951#1112315 (10hashar) [09:01:18] 10Continuous-Integration: Remove integration/kss.git - https://phabricator.wikimedia.org/T92482#1112322 (10hashar) 3NEW [09:04:06] 10Continuous-Integration, 10Wikimedia-Git-or-Gerrit: Gerrit: delete integration/kss.git repo - https://phabricator.wikimedia.org/T92483#1112334 (10hashar) 3NEW [09:09:32] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:18:37] 10Beta-Cluster: Diamond logstash monitor fills /var/log/apache2 access log - https://phabricator.wikimedia.org/T74175#1112398 (10hashar) 5Resolved>3Open Reopening, that is still happening. The monitoring is using the vhost `logstash.beta.wmflabs.org` thus the spam ends up in `/var/log/apache2/other_vhosts_ac... [09:21:00] aharoni: I am available in our usual time today, in 40 minutes [09:21:12] good thank you [09:21:20] aharoni: can you pair then? [09:21:52] yes [09:23:13] aharoni: great, see you then [09:24:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:55] 10Continuous-Integration, 10Wikimedia-Git-or-Gerrit: Gerrit: delete integration/kss.git repo - https://phabricator.wikimedia.org/T92483#1112488 (10QChris) I deleted the repo from gerrit itself. But since I am lacking acces to the replication targets, @demon can you remove it from github & co? [10:31:48] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:41:09] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [10:43:57] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [10:55:34] 10Continuous-Integration, 10Wikidata, 3§ Wikidata-Sprint-2015-02-03, 3§ Wikidata-Sprint-2015-02-25, 3§ Wikidata-Sprint-2015-03-11: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#1112565 (10adrianheine) Two of the last three qunit jobs failed with databas... [11:01:17] (03PS8) 10Hashar: Package python deps with dh-virtualenv [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195272 (https://phabricator.wikimedia.org/T48552) [11:13:55] 10Continuous-Integration, 7Technical-Debt, 7Tracking: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#1112590 (10adrianheine) [11:29:50] 10Deployment-Systems, 5Patch-For-Review: LocalisationUpdate broken since 2014-12-16 - https://phabricator.wikimedia.org/T85790#1112617 (10Se4598) any update here? I suspect it is still not working. [11:32:14] 10Continuous-Integration, 7Jenkins: PHP fatal errors are not visible in jenkins output in Wikibase phpunit jobs - https://phabricator.wikimedia.org/T92397#1112624 (10daniel) @legoktm so maybe it's an issue with the php output buffer? I would expect php to discard any buffers before reporting a fatal, but who k... [12:02:57] aharoni: we can not hear or see you [12:04:10] :( [12:04:20] I can see your photos, but I cannot hear anything. [12:05:47] aharoni: we still can not hear you [12:05:59] what nonsense :( [12:06:15] I tried from another device. [12:08:02] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-salt is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:08:15] zeljkof: grrrrrrrrrrrr [12:08:34] I tried three devices and you cannot hear me in any of them. [12:09:23] aharoni: could you please join our hangout from earlier today? [12:09:44] I will join you there, to see if this is because lu has created the hangout* [12:19:33] (03PS9) 10Hashar: Package python deps with dh-virtualenv [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195272 (https://phabricator.wikimedia.org/T48552) [12:31:00] I’m back [12:31:02] let’s fix thisss [12:31:06] this -> scap [12:39:45] Project beta-scap-eqiad build #44925: FAILURE in 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/44925/ [13:09:24] 10Continuous-Integration, 10Wikidata, 3§ Wikidata-Sprint-2015-02-25, 3§ Wikidata-Sprint-2015-03-11: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#1112805 (10JanZerebecki) How do you know they failed because of the database locked errors? They too show the... [13:37:07] 10Continuous-Integration, 10Wikimedia-Git-or-Gerrit: Gerrit: delete integration/kss.git repo - https://phabricator.wikimedia.org/T92483#1112885 (10demon) Deleted from github and gitblit. I don't have rights to delete here on Phab so @chasemp can you `./bin/remove destroy rCIKS` for us? [13:41:26] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:41:31] 10Deployment-Systems, 5Patch-For-Review: LocalisationUpdate broken since 2014-12-16 - https://phabricator.wikimedia.org/T85790#1112901 (10Nikerabbit) I heve seen patches in Gerrit trying to work out the permissions issues. Someone more close to those is needed to give more detailed status update. [13:46:26] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:56:46] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:50:50] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #184: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/184/ [14:53:51] thcipriani: hi [14:54:00] howdy [14:54:06] thcipriani: so that patch has been beaten into submission, and is live on deployment-prep now :) [14:54:12] and scap works... [14:54:30] and mwdeploys no longer reliant on NFS for keys :D [14:54:32] wait do you mean on staging? [14:55:56] thcipriani: no deployment-prep [14:56:12] thcipriani: now I need to merge it and make sure it doesn’t break anything on prod... [14:56:16] and then we get back to staging [14:56:25] so now there are 3 environs to worry about, deployment-prep, staging and prod... [14:56:56] 10Deployment-Systems, 6Release-Engineering: Clean up erroneously created wmf/1.20wmf21 branches - https://phabricator.wikimedia.org/T92501#1113097 (10Krinkle) 3NEW a:3mmodell [14:57:23] 10Deployment-Systems, 6Release-Engineering: Clean up erroneously created wmf/1.20wmf21 branches - https://phabricator.wikimedia.org/T92501#1113105 (10Krinkle) [14:57:26] YuviPanda: gotcha. Should I be testing patches on deployment-prep before moving over to staging? [15:01:21] thcipriani: hmm, probably. or at least see if it breaks... [15:01:39] thcipriani: another difference is that staging also requires the first time setup to be puppetized [15:01:42] deplyment prep doesn't [15:02:57] thcipriani: besdies, problems deployment-prep faces now would also be problems staging faces soon enough... [15:04:09] YuviPanda: sure. Being trying to be mindful about not actually "changing" anything. deployment-prep would be a good test. I'm not sure about the process for patching things there—is there anyone I should ping before making changes there? [15:04:31] thcipriani: I usually !log here, and make sure to cherry-pick only as long as necessary... [15:04:47] thcipriani: and if you’re anticipating major downtime, poking chrismcmahonbrb and greg-g would be good [15:05:13] * thcipriani nods [15:05:50] !log jenkins loves german again [15:05:56] Logged the message, Master [15:07:46] 10Deployment-Systems, 5Patch-For-Review: LocalisationUpdate broken since 2014-12-16 - https://phabricator.wikimedia.org/T85790#1113145 (10bd808) We had a successful manual run in the early morning (UTC) of 2015-03-12. If the normal cron job is successful on 2015-03-13 we may be able to declare victory. Attempt... [15:07:57] thcipriani: can you review https://gerrit.wikimedia.org/r/#/c/195340/? [15:08:31] YuviPanda: yup, I'll take a look. [15:15:12] (03PS1) 10Krinkle: Remove MediaWiki REL1_21 and REL1_22 triggers [integration/config] - 10https://gerrit.wikimedia.org/r/196234 [15:19:09] 10Continuous-Integration, 10Wikidata, 3§ Wikidata-Sprint-2015-02-25, 3§ Wikidata-Sprint-2015-03-11: fix the qunit tests for wikidata: mwext-Wikibase-qunit - https://phabricator.wikimedia.org/T74184#1113158 (10adrianheine) Sorry. It's way done in the mw-debug-www.log build artifact. [15:21:17] thcipriani: cool. we have like.. a 6h window starting now, and I’m wondering if we should just do it. [15:21:59] (03CR) 10Krinkle: [C: 032] Remove MediaWiki REL1_21 and REL1_22 triggers [integration/config] - 10https://gerrit.wikimedia.org/r/196234 (owner: 10Krinkle) [15:22:29] !log Fix Jenkins UI (was stuck in German) [15:22:32] Logged the message, Master [15:22:35] YuviPanda: everything is looking sane so far. Probably best to break stuff when you have more time to fix. [15:22:58] thcipriani: 6h seems plenty... [15:28:39] (03Merged) 10jenkins-bot: Remove MediaWiki REL1_21 and REL1_22 triggers [integration/config] - 10https://gerrit.wikimedia.org/r/196234 (owner: 10Krinkle) [15:29:37] (03PS1) 10Krinkle: Don't run mediawiki-core-bundle-rubocop on <= REL1_23 [integration/config] - 10https://gerrit.wikimedia.org/r/196239 [15:29:47] (03CR) 10Krinkle: [C: 032] Don't run mediawiki-core-bundle-rubocop on <= REL1_23 [integration/config] - 10https://gerrit.wikimedia.org/r/196239 (owner: 10Krinkle) [15:31:01] (03Merged) 10jenkins-bot: Don't run mediawiki-core-bundle-rubocop on <= REL1_23 [integration/config] - 10https://gerrit.wikimedia.org/r/196239 (owner: 10Krinkle) [15:31:03] !log Reloading Zuul to deploy Ia289ebb0 [15:31:07] Logged the message, Master [15:34:21] 10Deployment-Systems, 6Release-Engineering: Clean up erroneously created wmf/1.20wmf21 branches - https://phabricator.wikimedia.org/T92501#1113200 (10greg) p:5Triage>3Normal [15:35:08] YuviPanda: what has to be done to update all the beta scap targets? [15:35:31] thcipriani: uncheck the role::beta::scap_target (or whatever that was) and add beta::deploymentaccess [15:35:32] how manual is that going to be? [15:35:35] thcipriani: I’ve already done that [15:35:41] there were only 5 deployment targets [15:35:41] ah, cool. [15:41:38] 6Release-Engineering: Remove EOL MediaWiki release branches - https://phabricator.wikimedia.org/T92503#1113228 (10Krinkle) 3NEW [15:43:23] (03CR) 1020after4: [C: 032] move auth to an untracked file, auth.php [tools/release] - 10https://gerrit.wikimedia.org/r/195941 (owner: 1020after4) [15:43:50] (03PS1) 10Krinkle: Update mediawiki-core jobs to use standard jsduck [integration/config] - 10https://gerrit.wikimedia.org/r/196245 [15:44:27] (03PS2) 1020after4: migrate-patch utility to retain security patches [tools/release] - 10https://gerrit.wikimedia.org/r/195942 [15:46:33] greg-g: so… looks like there’s a 5hour window now, and I”d rather do it now than tomorrow. is that ok? [15:46:39] it has a +1 from thcipriani :) [15:47:18] YuviPanda: kk, I might have to mute wmf-insecte if the browser tests all start failing in the new couple hours (which is fine, unfortunately) [15:47:54] greg-g: this will affect prod, more like, hence :) [15:47:56] alright [15:48:17] oh right, duh [15:48:24] * greg-g hasn't finished first cup of coffee yet [15:48:30] doit! [15:50:05] (03PS2) 1020after4: Fix branched sub-submodule support [tools/release] - 10https://gerrit.wikimedia.org/r/195972 [15:51:17] (03CR) 1020after4: [C: 032] Fix branched sub-submodule support [tools/release] - 10https://gerrit.wikimedia.org/r/195972 (owner: 1020after4) [15:53:13] greg-g: DOING [15:53:39] * thcipriani holds on to butts [15:55:51] greg-g: thcipriani YAY IT WAS A NOOP IN PROD [15:55:54] AS IT SHOULD HAVE BEEN [15:55:58] :D :D [15:56:19] YuviPanda: \o/ nice! [15:57:32] I love those moments when you plan so much, worry so much, tremble and you hit enter, and then..... exactly as it should have gone. [15:57:46] s/tremble and/tremble as/ [15:58:13] :D [15:58:14] yeah [15:58:45] greg-g: beta is also closer to prod in so many ways now. [15:58:53] partly by moving beta, but also by moving prod [15:58:55] so that’s nice [16:02:16] :) [16:09:10] 10Continuous-Integration, 10Quality-Assurance, 6Release-Engineering, 7Browser-Tests: browsertest jobs should not be allowed to run for 10 hours - https://phabricator.wikimedia.org/T92275#1113352 (10zeljkofilipin) The three longest running [[ https://integration.wikimedia.org/ci/view/BrowserTests/view/-All/... [16:10:42] 6Release-Engineering: Remove EOL MediaWiki release branches - https://phabricator.wikimedia.org/T92503#1113365 (10greg) p:5Triage>3Normal [16:13:59] (03CR) 10Krinkle: [C: 032] "Deployed mediawiki-core-jsduck-publish and mediawiki-core-jsduck." [integration/config] - 10https://gerrit.wikimedia.org/r/196245 (owner: 10Krinkle) [16:15:41] 10Continuous-Integration, 10Quality-Assurance, 6Release-Engineering, 7Browser-Tests: browsertest jobs should not be allowed to run for 10 hours - https://phabricator.wikimedia.org/T92275#1113395 (10greg) >>! In T92275#1113352, @zeljkofilipin wrote: > I think it is safe to set the limit to 3 hours for all b... [16:18:06] 10Continuous-Integration, 10Quality-Assurance, 6Release-Engineering, 7Browser-Tests: browsertest jobs should not be allowed to run for 10 hours - https://phabricator.wikimedia.org/T92275#1113403 (10zeljkofilipin) >>! In T92275#1105133, @hashar wrote: > Can we look at the reason for the slowness? Maybe we a... [16:18:23] 10Continuous-Integration, 10Quality-Assurance, 6Release-Engineering, 7Browser-Tests: browsertest jobs should not be allowed to run for 10 hours - https://phabricator.wikimedia.org/T92275#1113404 (10zeljkofilipin) a:3zeljkofilipin [16:19:33] (03Merged) 10jenkins-bot: Update mediawiki-core jobs to use standard jsduck [integration/config] - 10https://gerrit.wikimedia.org/r/196245 (owner: 10Krinkle) [16:19:38] 10Continuous-Integration, 10Quality-Assurance, 6Release-Engineering, 7Browser-Tests: browsertest jobs should not be allowed to run for 10 hours - https://phabricator.wikimedia.org/T92275#1104513 (10zeljkofilipin) >>! In T92275#1113395, @greg wrote: >>>! In T92275#1113352, @zeljkofilipin wrote: >> I think i... [16:22:13] 6Release-Engineering, 7Browser-Tests: Investigate updating browser versions in Jenkins builds. - https://phabricator.wikimedia.org/T92005#1113423 (10zeljkofilipin) Wait, I am not sure what is the problem here. The first job is a Firefox job, the second job is Chrome job. Firefox job failed maybe because Flow... [16:43:41] Krinkle|detached: hashar where will I find the definitions for the beta scap job? [16:43:50] it’s not in the jenkins-job-builder repo [16:43:55] yes it is [16:44:27] bd808: hmm, last commit on the repo seems to be from… October? [16:44:31] YuviPanda: integration/config [16:44:32] https://github.com/wikimedia/integration-config/blob/master/jjb/beta.yaml [16:44:32] I think I’m looking at the wrong one [16:44:33] gah [16:44:34] right [16:44:53] jjb is just our fork/mirror of the code [16:45:00] https://github.com/wikimedia/integration-config/blob/master/jjb/beta.yaml#L210-L240 [16:45:19] but yeah that's not in the jjb repo any more [16:45:23] bd808: btw, I merged the patch :D beta is using keyholder now, and there are no more keys in NFS for mwdeploy, and best of all, it was a no-op on prod :D [16:45:34] perfect! [16:45:51] thanks to ori for making that possible! [16:45:59] bd808: yup! :D three days and lots of shaven yaks. [16:46:16] if I start thanking ori for all the things he’s made possible I’ll be thanking ori all day :P [16:46:22] same for you as well [16:47:06] says the guy who has infrastructure named after him ;) [16:58:23] bd808: vagrant you mean? :) [16:58:34] heh [17:03:57] 6Release-Engineering, 10Staging, 3releng-201415-Q3: Determine code update cycle/cadence for the staging cluster - https://phabricator.wikimedia.org/T91563#1113560 (10mmodell) Why not just have staging-tin track mediawiki head and automatically run scap every time something changes? [17:07:29] 6Release-Engineering, 10Staging, 3releng-201415-Q3: Determine code update cycle/cadence for the staging cluster - https://phabricator.wikimedia.org/T91563#1113567 (10greg) >>! In T91563#1113560, @mmodell wrote: > Why not just have staging-tin track mediawiki head and automatically run scap every time somethi... [17:08:48] 10Deployment-Systems, 5Patch-For-Review: l10nupdate user can't access scap shared ssh key causing nightly l10nupdate sync process to fail - https://phabricator.wikimedia.org/T76061#1113586 (10mmodell) a:5mmodell>3bd808 looks like you have this one handled. [17:13:31] uh oh [17:13:53] bd808: scap question... [17:14:15] bd808: in prod, scap doesn’t run *as* mwdeploy, right? on the deployment host, when someone invokes it? It only sshs to instances as mwdeploy... [17:14:37] it runs as mwdeploy [17:14:47] most of the time [17:15:14] does things like "sudo -u mwdeploy rsync ..." [17:15:31] oh on the deploy host.... [17:15:34] yeah not in prod [17:15:52] it runs as whoever types "scap" [17:16:18] anyone in the wikidev group [17:16:57] but it runs as mwdeploy on the target hosts [17:17:19] even when it connects over ssh as another user [17:17:40] bd808: right, that was my thinking as well. [17:18:06] so now things are a bit eugh on beta, since keyholder is accessible only to project-deployment-prep, and jekins-bot isn’t project-deployment-prep... [17:18:38] * YuviPanda thinks [17:18:55] oooh [17:18:58] jenkins-deploy? [17:19:03] make the key available to another group? [17:19:17] jenkins-deploy is a proper member... [17:19:33] is that the user that kicks off scap from jenkins? [17:19:40] *shrug* [17:19:47] yeah, I’ll poke [17:20:09] bd808: ideall, I’ll shave this yak too, although I’m mostly afraid of touching anything CI... [17:20:14] so maybe not. [17:20:19] thanks though :) [17:20:35] 10Continuous-Integration, 10Wikidata, 7Jenkins: PHP fatal errors are not visible in jenkins output in mwext-Wikibase-client-tests job - https://phabricator.wikimedia.org/T92397#1113674 (10Legoktm) [17:23:09] 10Continuous-Integration, 10Wikidata, 7Jenkins: PHP fatal errors are not visible in jenkins output in mwext-Wikibase-client-tests job - https://phabricator.wikimedia.org/T92397#1113684 (10Legoktm) Hmm, is something in client messing with the output buffer? I grepped for "ob_" but didn't see anything client-r... [17:25:36] hashar: around/ [17:25:49] hashar: do you know what user jenkins uses to ssh to deployment-bastion? [17:26:14] 10Continuous-Integration, 6Scrum-of-Scrums, 7Blocked-on-MediaWiki-Core: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1113698 (10Krinkle) 5Resolved>3Open https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qun... [17:26:14] 10Continuous-Integration, 10VisualEditor, 7Tracking, 3VisualEditor 2014/15 Q3 blockers: Browser disconnects when running QUnit tests with local browsers (tracking) - https://phabricator.wikimedia.org/T89075#1113700 (10Krinkle) [17:27:29] 10Continuous-Integration, 6Scrum-of-Scrums, 7Blocked-on-MediaWiki-Core: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1113705 (10Krinkle) a:5Krinkle>3None [17:30:04] 10Continuous-Integration, 10Flow, 6Mobile-Web, 10VisualEditor, 10Wikimedia-Hackathon-2015: Create Jenkins builds for Editing across repositories (MobileFrontend, VisualEditor etc) - https://phabricator.wikimedia.org/T90647#1113707 (10Jdforrester-WMF) p:5Triage>3Normal [17:30:36] 6Release-Engineering, 10VisualEditor, 7Browser-Tests: Create VisualEditor tests targeting the older version of browsers (Chrome and Firefox for now) for better backward compatibility - https://phabricator.wikimedia.org/T90678#1113710 (10Jdforrester-WMF) p:5Triage>3High [17:30:45] 6Release-Engineering, 10VisualEditor, 7Browser-Tests: Selenium bug with Firefox causes VE test failure - https://phabricator.wikimedia.org/T90651#1113712 (10Jdforrester-WMF) p:5Triage>3High [17:31:24] (03PS1) 10Yuvipanda: Don't use separate ssh-agent for running scap [integration/config] - 10https://gerrit.wikimedia.org/r/196273 [17:31:56] legoktm: hashar ^ should unbreak scap on beta. help? [17:32:39] thcipriani: ^d I am considering just getting rid of deployment-rsync01... [17:32:48] 6Release-Engineering, 10VisualEditor, 7Browser-Tests: Selenium bug with Firefox causes VE test failure - https://phabricator.wikimedia.org/T90651#1113724 (10Cmcmahon) Note that I removed the "&" from the string with this commit: https://gerrit.wikimedia.org/r/#/c/195683 [17:33:04] <^d> YuviPanda: No objections here [17:33:13] twentyafterfour: ^ [17:34:18] looking [17:34:20] ? [17:35:24] twentyafterfour: getting rid of deployment-rscync01 [17:35:58] YuviPanda: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/44941/console [17:36:16] YuviPanda: that's with your change, still failing [17:36:30] legoktm: yes, that’s rsync01, that I want to delete [17:36:36] legoktm: before that it was *all* failing [17:36:45] legoktm: let me find a bug to associate with it [17:36:52] actually fuck it :) [17:36:56] legoktm: but yeah, that output is desirable [17:36:59] ok [17:37:26] YuviPanda: why are you killing it? [17:37:36] thcipriani: why is it there? [17:37:42] thcipriani: in prod, we don’t have separate proxy servers [17:37:46] some mw servers are also proxy servers [17:37:59] (03CR) 10Legoktm: [C: 032] "Deployed and confirmed: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/44941/console" [integration/config] - 10https://gerrit.wikimedia.org/r/196273 (owner: 10Yuvipanda) [17:38:06] so scap will fail here, because rsync01 doesn’t actually have any mediawiki in it [17:38:19] thcipriani: plus, it’s causing the job to fail ;D [17:38:43] by all means, kill it [17:38:46] YuviPanda: that sounds like justification enough for me. [17:38:54] +1 works for me [17:38:57] wheeeee [17:39:34] !log killll deployment-rsync01, wasn’t being used for anything discernable, and that’s not how proxies work in prod [17:39:38] Logged the message, Master [17:40:29] rsync01 was handy when I was figuring out how to bring scap into beta but I agree it doesn't help anything now [17:41:13] \o/ [17:41:14] cool [17:41:58] bd808: with that, I wonder if I can declare that prod and beta have same deployment setup going... [17:42:04] or if there are wast swathes of stuff I’m missing [17:43:00] the only thing I can think of is the hack in scap for beta to ignore host keys [17:43:23] bd808: aah, right. [17:43:23] that [17:43:27] (03Merged) 10jenkins-bot: Don't use separate ssh-agent for running scap [integration/config] - 10https://gerrit.wikimedia.org/r/196273 (owner: 10Yuvipanda) [17:44:51] bd808: I want to kill ./files/scripts/beta-apaches as well. I think it’s trivial enough to be salted if needed... [17:44:58] I can keep it if you think it’s useful... [17:45:10] * YuviPanda is trying to pare down the beta module to as lean as possible [17:45:45] I don't use it [17:45:57] it was an ask from hashar in the past [17:46:12] oh, hmm. [17:48:23] bd808: also [17:48:24] class role::beta::trebuchet_testing { [17:48:25] that? [17:50:59] YuviPanda: not sure. what hosts is it applied on? [17:52:40] bd808: hmm, none that I can find, actually [17:52:57] then kill with fire :) [17:54:34] deployment-db1 has it... [17:54:38] wat. [17:54:41] ... [17:54:53] why would deployment-db1 have… trebuchet_testing [17:55:32] deployment-mathoid and deployment-rsync01...I guess..._had_ it [17:55:49] deployment-mathoid _has_ it [17:56:03] it might be something ariel was playing with at some point [17:56:11] apergos: ^ [17:56:37] Yippee, build fixed! [17:56:37] Project beta-scap-eqiad build #44946: FIXED in 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/44946/ [17:56:50] legoktm: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/44946/console scap success \o/ [17:57:05] I don’t think I’ve ever felt as good about a green before. [17:57:13] it's just for testing [17:57:13] YuviPanda: https://github.com/wikimedia/operations-puppet/commit/22857337f3f17433e8f04d518129fdae672b1bfa [17:57:19] nothing else, so please don't kill it [17:57:33] apergos: hmm, what is it testing? [17:57:37] trebuchet! [17:57:41] * YuviPanda is trying to move stuff off beta into places elsewhere. [17:57:54] hmm [17:58:03] well this is stuff I want in beta because that's the good subset to test trebuchet out after salt [17:58:30] I want a little cluster with a few different oses that looks mostly like prod etc, and it fits the bill [17:58:33] apergos: right, having it be in beta is completely fine. [17:58:37] ok [17:58:38] apergos: I just want to move the *code* [17:58:41] ahhh [17:58:44] :-D [17:58:54] like, all the scap code just got moved / merged with scap/ module [17:58:59] oh I see [17:59:00] YuviPanda: yay! [17:59:09] legoktm: it’s running scap the same was as prod now :) [18:00:05] I'm not coming up with any bright ideas [18:00:28] yeah, me neither. [18:00:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:02:15] uh [18:02:23] transient? [18:02:25] * YuviPanda looks [18:05:08] seems to be [18:05:31] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [18:09:52] that’s better [18:10:14] thcipriani: so for staging-tin we still have to do a lot of ‘first time setup steps’ in puppet. like cloning the mediawiki repo, cloning mediawiki-config repo, etc [18:10:29] thcipriani: and there are going to be some manual steps (like setting up db password?) [18:10:36] not sure how to handle the manual steps [18:11:14] YuviPanda: for mariadb, right now, I've just got a bash script that is nowhere :\ so I'm not sure either [18:12:18] thcipriani: I suppose we can start a document of ‘how to build yourself a beta cluster’, which has links to these scripts (which can be put in puppet) [18:14:22] YuviPanda: that sounds good to me. [18:14:40] thcipriani: cool :) [18:24:20] Krinkle: are you fine with ori his request to remove $wgResourceLoaderMaxQueryLength and hardcode the default?: https://gerrit.wikimedia.org/r/#/c/193130/ [18:24:27] ^d, I think this causes the 5::duration bug: http://en.wikipedia.org/w/api.php?action=query&list=recentchanges&rcprop=loginfo&rctype=log&rcstart=20150312084618 [18:24:37] log_id = 63139902 [18:25:03] log_params is just indefinite\nnocreate [18:25:24] ApiQueryRecentChanges does not pass $legacy [18:25:49] 10Continuous-Integration, 6MediaWiki-Core-Team, 6operations: add a check for whitespace before leading so it defaults to assuming not legacy and tries to access 5::duration, when we actually want 0 [18:26:11] ApiQueryWatchlist has a similar issue [18:27:10] 10Continuous-Integration, 6MediaWiki-Core-Team, 6operations: add a check for whitespace before leading Krenair: Ah, I see it now [18:28:36] do we know why legacy logs are still being recorded though? [18:30:43] 10Deployment-Systems: scap's check_php_syntax() should check for text before ' 10Continuous-Integration, 6MediaWiki-Core-Team, 6operations: add a check for whitespace before leading >! In T92531#1113969, @Krinkle wrote: > See {T46875}. > > This and various other errors are already caught by phpcs. Please ensure proj... [18:32:49] 10Continuous-Integration, 6MediaWiki-Core-Team, 6operations: add a check for whitespace before leading >! In T92531#1113969, @Krinkle wrote: > See {T46875}. > > This and various other errors are already caught by phpcs. Please ensure project(... [18:33:02]