[03:17:52] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #767: 04FAILURE in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/767/ [03:26:31] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<11.11%) [04:12:55] 10Beta-Cluster-Infrastructure: Twinkle menu title letters distorted - https://phabricator.wikimedia.org/T117384#1772510 (10Skyllfully) 3NEW [04:27:25] 5Gerrit-Migration, 6Release-Engineering-Team, 15User-Bd808-Test: Setup differential for https://github.com/wikimedia/composer-merge-plugin - https://phabricator.wikimedia.org/T117293#1772523 (10mmodell) @bd808: This is an excellent idea. I've created rGCMP which should be mirroring from github. Now all you n... [04:28:06] 5Gerrit-Migration, 6Release-Engineering-Team, 15User-Bd808-Test: Setup differential for https://github.com/wikimedia/composer-merge-plugin - https://phabricator.wikimedia.org/T117293#1772526 (10mmodell) p:5Triage>3Normal [04:33:37] 5Gerrit-Migration, 6Release-Engineering-Team, 15User-Bd808-Test: Setup differential for https://github.com/wikimedia/composer-merge-plugin - https://phabricator.wikimedia.org/T117293#1772529 (10Legoktm) is Facebook/HHVM's pull request importer bot open source? [04:43:35] 10MediaWiki-Releasing, 6Release-Engineering-Team, 10MediaWiki-General-or-Unknown: Remove EOL MediaWiki release branches - https://phabricator.wikimedia.org/T92503#1772531 (10mmodell) @hashar: why wouldn't tags work for all of those purposes? [04:47:05] 10Differential, 6Release-Engineering-Team, 15User-Bd808-Test: Setup differential for https://github.com/wikimedia/composer-merge-plugin - https://phabricator.wikimedia.org/T117293#1772541 (10greg) [05:12:10] 10Differential, 6Release-Engineering-Team, 15User-Bd808-Test: Setup differential for https://github.com/wikimedia/composer-merge-plugin - https://phabricator.wikimedia.org/T117293#1772571 (10bd808) >>! In T117293#1772529, @Legoktm wrote: > is Facebook/HHVM's pull request importer bot open source? I haven't... [05:27:09] 10Differential, 6Release-Engineering-Team, 15User-Bd808-Test: Setup differential for https://github.com/wikimedia/composer-merge-plugin - https://phabricator.wikimedia.org/T117293#1772580 (10bd808) [05:28:45] 10Differential, 6Release-Engineering-Team, 15User-Bd808-Test: Setup differential for https://github.com/wikimedia/composer-merge-plugin - https://phabricator.wikimedia.org/T117293#1772582 (10mmodell) I guess I need this to merge before phabricator will mirror from github: https://gerrit.wikimedia.org/r/#/c/2... [05:29:26] Yippee, build fixed! [05:29:26] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #590: 09FIXED in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/590/ [05:39:50] Yippee, build fixed! [05:39:50] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #241: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/241/ [06:36:29] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [09:00:35] 10Beta-Cluster-Infrastructure: Twinkle menu title letters distorted - https://phabricator.wikimedia.org/T117384#1772775 (10hashar) The Twinkle Gadget on enwiki beta cluster is from 2011 : http://en.wikipedia.beta.wmflabs.org/wiki/MediaWiki:Gadget-Twinkle So it probably needs to be updated :-} Issues can be re... [09:15:41] (03PS1) 10Hashar: [WMF] pin python-jenkins <0.4.9 [integration/jenkins-job-builder] - 10https://gerrit.wikimedia.org/r/250381 [09:16:42] (03CR) 10Hashar: [C: 032 V: 032] "Force merge for our fork. Whenever we upgrade JJB, we will have to force push to override that local hack." [integration/jenkins-job-builder] - 10https://gerrit.wikimedia.org/r/250381 (owner: 10Hashar) [09:19:57] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #1: 04FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/1/ [09:31:23] No such file or directory - ../../i18n/.json [09:31:25] :^ [09:54:47] PROBLEM - Puppet staleness on integration-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [10:32:36] 10Beta-Cluster-Infrastructure: Twinkle menu title letters on Beta Simple English Wikipedia distorted - https://phabricator.wikimedia.org/T117384#1772941 (10Aklapper) p:5Normal>3Triage [10:33:13] 10Beta-Cluster-Infrastructure, 6Community-Tech: Twinkle menu title letters on Beta Simple English Wikipedia distorted - https://phabricator.wikimedia.org/T117384#1772510 (10Aklapper) ([[https://www.mediawiki.org/wiki/Phabricator/Project_management#Setting_task_priorities | Resetting task priority to default]]) [10:52:22] !log Restarting Jenkins to upgade Gearman plugin from 0.1.2 to 0.1.3 "Send node labels back on build completion" [10:52:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:59:28] !log Bump integration/zuul upstream branch from 3ebedde to 1cc37f7 [10:59:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:21:03] (03PS1) 10Hashar: Bump Debian version and rebase patches [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/250396 [11:52:10] 10Continuous-Integration-Config, 10Analytics, 6WMDE-Analytics-Engineering, 10Wikidata: Add basic jenkins linting to analytics-limn-wikidata-data - https://phabricator.wikimedia.org/T116007#1773147 (10Addshore) [11:59:57] (03PS1) 10JanZerebecki: Add repo analytics/limn-wikidata-data [integration/config] - 10https://gerrit.wikimedia.org/r/250399 (https://phabricator.wikimedia.org/T116007) [12:06:40] (03PS2) 10Hashar: Bump Debian version and rebase patches [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/250396 (https://phabricator.wikimedia.org/T98294) [12:12:02] (03CR) 10Hashar: "Build result at https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-60-g1cc37f7-wmf1precise1/" [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/250396 (https://phabricator.wikimedia.org/T98294) (owner: 10Hashar) [12:14:31] (03CR) 10Addshore: [C: 031] Add repo analytics/limn-wikidata-data [integration/config] - 10https://gerrit.wikimedia.org/r/250399 (https://phabricator.wikimedia.org/T116007) (owner: 10JanZerebecki) [12:14:44] 10Continuous-Integration-Config, 10Analytics, 6WMDE-Analytics-Engineering, 10Wikidata, 5Patch-For-Review: Add basic jenkins linting to analytics-limn-wikidata-data - https://phabricator.wikimedia.org/T116007#1773176 (10Addshore) a:3JanZerebecki [12:17:33] 10Continuous-Integration-Infrastructure, 5Patch-For-Review, 7Zuul: Bump python-gear package to 0.5.7 - https://phabricator.wikimedia.org/T98294#1773180 (10hashar) https://gerrit.wikimedia.org/r/250396 causes the Zuul Debian package on Precise to embed Gear 0.5.8. [12:24:13] !log unpooling integration-slave-precise1013 and upgrading zuul-cloner 2.0.0-327-g3ebedde-wmf3precise1 -> /root/zuul_2.1.0-60-g1cc37f7-wmf1precise1_amd64.deb [12:24:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:27:18] !log Upgraded zuul-cloner on Precise slaves: 2.0.0-327-g3ebedde-wmf3precise1 -> /root/zuul_2.1.0-60-g1cc37f7-wmf1precise1_amd64.deb [12:27:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:27:27] PROBLEM - salt-minion processes on scandium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [12:35:03] RECOVERY - salt-minion processes on scandium is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [12:37:32] (03CR) 10Hashar: [C: 032 V: 032] "Upgraded on Precise." [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/250396 (https://phabricator.wikimedia.org/T98294) (owner: 10Hashar) [12:38:50] 10Browser-Tests, 10Continuous-Integration-Infrastructure: Delete ruby2.0lint job and only run bundle-rubocop job for repositories with Ruby code - https://phabricator.wikimedia.org/T114262#1773214 (10zeljkofilipin) As far as I remember, during one of browser tests triage meetings, @hashar, @dduvall and me have... [12:42:17] 10Continuous-Integration-Config, 7Puppet: Move RuboCop job from experimental pipeline to the usual pipelines for operations/puppet - https://phabricator.wikimedia.org/T110019#1773216 (10zeljkofilipin) a:5zeljkofilipin>3None [12:50:38] 10Continuous-Integration-Config: Move Bundler Jenkins jobs to Nodepool instances - https://phabricator.wikimedia.org/T114860#1773231 (10zeljkofilipin) a:5zeljkofilipin>3None [13:06:29] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 10VisualEditor: browsertests-VisualEditor-language-screenshot-os_x_10.10-firefox jenkins job failing - https://phabricator.wikimedia.org/T111510#1773258 (10zeljkofilipin) [13:06:31] 10Browser-Tests, 10VisualEditor, 5Patch-For-Review: Delete or fix failed VisualEditor browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94162#1773259 (10zeljkofilipin) [13:06:33] 10Browser-Tests, 10VisualEditor, 5Patch-For-Review: No such file or directory - i18n/af.json (Errno::ENOENT) - https://phabricator.wikimedia.org/T111525#1773256 (10zeljkofilipin) 5Open>3Resolved Fixed in https://gerrit.wikimedia.org/r/#/c/247630/ [13:13:01] (03PS1) 10Hashar: zuul: sync with upstream @1cc37f7 [integration/docroot] - 10https://gerrit.wikimedia.org/r/250414 [13:13:55] 10Continuous-Integration-Config, 10Analytics, 6WMDE-Analytics-Engineering, 10Wikidata, and 2 others: Add basic jenkins linting to analytics-limn-wikidata-data - https://phabricator.wikimedia.org/T116007#1773275 (10JanZerebecki) [13:14:12] PROBLEM - salt-minion processes on scandium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [13:15:00] 10Browser-Tests, 10VisualEditor, 5Patch-For-Review: Delete or fix failed VisualEditor browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94162#1773277 (10zeljkofilipin) a:5zeljkofilipin>3None @vikassy has fixed the problem that caused all VisualEditor browser tests to fail (T111525). I have... [13:15:46] (03CR) 10Zfilipin: "I have left a comment at related task in Phabricator, it is easier to format and edit than here: T94162#1773277" [integration/config] - 10https://gerrit.wikimedia.org/r/247612 (https://phabricator.wikimedia.org/T94162) (owner: 10Jforrester) [13:17:52] (03CR) 10Hashar: [C: 032] "@Timo I have bumped Zuul to a more recent upstream version. Cherry picked that patch on gallium and it seems to work fine." [integration/docroot] - 10https://gerrit.wikimedia.org/r/250414 (owner: 10Hashar) [13:18:46] (03Merged) 10jenkins-bot: zuul: sync with upstream @1cc37f7 [integration/docroot] - 10https://gerrit.wikimedia.org/r/250414 (owner: 10Hashar) [13:18:53] (03CR) 10Paladox: "Needs re c+2 didn't start the merging process." [integration/docroot] - 10https://gerrit.wikimedia.org/r/250414 (owner: 10Hashar) [13:20:54] (03CR) 10Hashar: "Paladox what are you talking about?" [integration/docroot] - 10https://gerrit.wikimedia.org/r/250414 (owner: 10Hashar) [13:22:46] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 7Documentation: [Story] Automatically create Doxygen documentation of Wikibase PHP code - https://phabricator.wikimedia.org/T66996#1773292 (10Jonas) [13:23:35] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T94162 build #1: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce-T94162/1/ [13:25:52] (03CR) 10Paladox: "Sorry. I wrote this just before the status changed to merged." [integration/docroot] - 10https://gerrit.wikimedia.org/r/250414 (owner: 10Hashar) [13:30:43] !log updating Nodepool base image on wmflabs to get https://gerrit.wikimedia.org/r/#/c/250148/ (set hostname on debian hosts) [13:30:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:31:07] !log openstack image set --name ci-jessie-wikimedia_old_20151102 ci-jessie-wikimedia [13:31:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:31:38] openstack image create --file image-jessie-20151030T221648Z.qcow2 ci-jessie-wikimedia --disk-format qcow2 --property show=true [13:31:42] !log openstack image create --file image-jessie-20151030T221648Z.qcow2 ci-jessie-wikimedia --disk-format qcow2 --property show=true [13:31:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:32:46] !log Refreshing Nodepool snapshot to be based on new image image-jessie-20151030T221648Z.qcow2 : git -C /etc/nodepool/wikimedia/ pull && nodepool image-update wmflabs-eqiad ci-jessie-wikimedia [13:32:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:34:43] RECOVERY - salt-minion processes on scandium is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [13:41:28] 10Continuous-Integration-Config, 5Continuous-Integration-Scaling, 5Patch-For-Review, 7Upstream, 7WorkType-Maintenance: Nodeopool instances do not acquire hostname from cloud (always: 'debian') - https://phabricator.wikimedia.org/T117283#1773354 (10hashar) 5Open>3Resolved I have pushed the new image f... [13:42:02] RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 1.01 ms [13:45:36] 5Continuous-Integration-Scaling, 6operations: Allow network flow between labs instance and scandium - https://phabricator.wikimedia.org/T116975#1773382 (10hashar) a:5hashar>3chasemp That follows up a discussion with @chasemp . Not sure exactly who can handle the firewall rules. I would be happy to do a vi... [13:50:33] (03CR) 10Hashar: [C: 032] Add repo analytics/limn-wikidata-data [integration/config] - 10https://gerrit.wikimedia.org/r/250399 (https://phabricator.wikimedia.org/T116007) (owner: 10JanZerebecki) [13:51:55] (03Merged) 10jenkins-bot: Add repo analytics/limn-wikidata-data [integration/config] - 10https://gerrit.wikimedia.org/r/250399 (https://phabricator.wikimedia.org/T116007) (owner: 10JanZerebecki) [13:55:15] 10Continuous-Integration-Config, 10Analytics, 6WMDE-Analytics-Engineering, 10Wikidata, and 2 others: Add basic jenkins linting to analytics-limn-wikidata-data - https://phabricator.wikimedia.org/T116007#1773442 (10Addshore) 5Open>3Resolved [13:55:18] (03Abandoned) 10Hashar: Revert "Delete failing VisualEditor browsertests Jenkins jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/247612 (https://phabricator.wikimedia.org/T94162) (owner: 10Jforrester) [13:57:15] PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145) [13:59:18] 10Continuous-Integration-Config, 10Analytics: add CI for repos analytics/limn-*-data - https://phabricator.wikimedia.org/T117416#1773454 (10JanZerebecki) 3NEW [14:02:14] 10Continuous-Integration-Config, 10Analytics, 6WMDE-Analytics-Engineering, 10Wikidata, and 2 others: Add basic jenkins linting to analytics-limn-wikidata-data - https://phabricator.wikimedia.org/T116007#1773474 (10JanZerebecki) >>! In T116007#1757055, @hashar wrote: > Don't we want jobs for all the limn da... [14:12:29] 10Browser-Tests: make the selenium driver for chrome available - https://phabricator.wikimedia.org/T117418#1773487 (10JanZerebecki) 3NEW [14:30:47] 5Continuous-Integration-Scaling, 6operations: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1773519 (10fgiunchedi) ok I've uploaded `oslo-sphinx` `python-oslotest` and `openstack-pkg-tools` (build-dep for oslo-sphinx) to `jessie-wikimed... [14:44:52] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1773593 (10JanZerebecki) After looking at the results running without saucelabs: I filed: {T117418} The jobs run in about half the time. It seems the runs have... [14:55:33] 10Browser-Tests: make the selenium driver for chrome available - https://phabricator.wikimedia.org/T117418#1773651 (10zeljkofilipin) I was sure we already had this task, but I can not find it now. I think we already have Chromium available, but we need to let Selenium know where it is: https://github.com/watir/... [14:55:41] 5Continuous-Integration-Scaling, 6operations: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1773653 (10hashar) Nice, seems that fixed it :-} I backported python-os-client-config_1.6.0-4 using: ``` git clone git://anonscm.debian.org/o... [14:55:51] 10Browser-Tests: make the selenium driver for chrome available - https://phabricator.wikimedia.org/T117418#1773656 (10zeljkofilipin) p:5Triage>3Normal [15:04:54] 5Continuous-Integration-Scaling, 6operations: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1773704 (10hashar) Once done, we can backport `python-shade` (T107267) [15:08:45] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1773721 (10Jonas) Is this the right [[ http://raita.wmflabs.org/#projects/https%3A%2F%2Fgithub.com%2Fwmde%2FWikidataBrowserTests.git/builds/latest | link ]] to... [15:23:00] (03CR) 10Jforrester: "> I disagree that VisualEditor browser tests were fine before we deleted the jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/247612 (https://phabricator.wikimedia.org/T94162) (owner: 10Jforrester) [15:41:39] (03PS3) 10JanZerebecki: Additionally run Wikidata browsertests without saucelabs [integration/config] - 10https://gerrit.wikimedia.org/r/247901 (https://phabricator.wikimedia.org/T116166) [15:43:38] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1773861 (10JanZerebecki) For now lets run the jobs both with and without saucelabs until we figure out how to make one of them not flaky. [15:44:53] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review, 3Wikidata-Sprint-2015-10-13: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1773875 (10JanZerebecki) [15:48:26] 10Browser-Tests, 10MediaWiki-extensions-MultimediaViewer, 5Patch-For-Review: Add E2E test for "Expand" links on file pages - https://phabricator.wikimedia.org/T77373#1773889 (10zeljkofilipin) 5stalled>3declined a:3zeljkofilipin Looks like nobody plans to work no this. Please reopen if needed. [15:49:27] 10Browser-Tests, 10VisualEditor, 10VisualEditor-ContentEditable: VisualEditor: Inserting some content in a QA script doesn't round-trip (null diff) - failure to inserting CE content into DM somehow? - https://phabricator.wikimedia.org/T55721#1773897 (10zeljkofilipin) Is this still a problem, or can the task... [15:51:27] 10Browser-Tests, 10MediaWiki-extensions-UniversalLanguageSelector: Scenario fails on ciwmf: Display language change is previewed immediately - https://phabricator.wikimedia.org/T60031#1773904 (10zeljkofilipin) 5Open>3Invalid a:3zeljkofilipin Jenkins jobs running ULS browser tests have been deleted since... [15:52:12] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review, 3Wikidata-Sprint-2015-10-13: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1773909 (10Tobi_WMDE_SW) That's the nice thing about saucelabs: you can have a look at the test-run in a video to figure out what... [15:52:50] 10Browser-Tests, 10MediaWiki-User-login-and-signup: browser readyState should be "loading" until login completes - https://phabricator.wikimedia.org/T64001#1773911 (10zeljkofilipin) 5Open>3Invalid a:3zeljkofilipin I do not remember seeing problems with logging in for a long time. Please reopen if needed. [15:54:46] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow: Flow browser test: hover failure in IE in flow_logged_in test - https://phabricator.wikimedia.org/T69723#1773920 (10zeljkofilipin) Is this still a problem, or can this task be resolved? [15:56:29] 10Browser-Tests, 10Wikidata: No test report files were found: job fails in jenkins but is shown as successful in raita - https://phabricator.wikimedia.org/T116164#1773925 (10zeljkofilipin) p:5Triage>3Normal [15:56:40] 10Browser-Tests: jenkins doesn't show the real failed tests, but raita does - https://phabricator.wikimedia.org/T116162#1773928 (10zeljkofilipin) p:5Triage>3Normal [15:56:42] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review, 3Wikidata-Sprint-2015-10-13: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1773927 (10JanZerebecki) If you can figure that out, then what does go wrong with the last few failures on sauce labs? [15:56:52] 10Browser-Tests: browsertest failure reports don't show the failing tests saucelabs link, but a different one - https://phabricator.wikimedia.org/T115500#1773929 (10zeljkofilipin) p:5Triage>3Normal [15:57:06] 10Browser-Tests, 10Continuous-Integration-Infrastructure: Delete ruby2.0lint job and only run bundle-rubocop job for repositories with Ruby code - https://phabricator.wikimedia.org/T114262#1773931 (10zeljkofilipin) p:5Triage>3Normal [15:57:27] (03PS1) 10Hashar: Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/250442 [15:58:25] 10Deployment-Systems, 3Scap3, 7Documentation: Regenerate scap documentation when using differential for code review - https://phabricator.wikimedia.org/T117431#1773933 (10thcipriani) 3NEW [15:59:14] 10Browser-Tests, 10MediaWiki-extensions-GettingStarted: Browser tests for guided tours being delivered by the GettingStarted toolbar - https://phabricator.wikimedia.org/T54246#1773942 (10zeljkofilipin) Does this still need to be implemented, or can this task be resolved? [16:01:28] 10Browser-Tests, 10MediaWiki-extensions-Translate: Can not make a translation - https://phabricator.wikimedia.org/T68415#1773946 (10zeljkofilipin) 5Open>3Invalid a:3zeljkofilipin Jenkins jobs running browser tests for Translate extension are deleted, since they were broken. https://integration.wikimedia... [16:04:00] (03PS2) 10Hashar: Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/250442 [16:05:08] 10Browser-Tests, 10MediaWiki-extensions-MultimediaViewer: (possibly spurious) "Could not load image data http" errors on Beta Cluster - https://phabricator.wikimedia.org/T74162#1773955 (10zeljkofilipin) 5Open>3Invalid a:3zeljkofilipin I do not remember seeing this problem in a long time. Please reopen if... [16:06:23] 10Browser-Tests, 10MediaWiki-Page-editing, 7Browser-Support-Internet-Explorer: IE7 and IE8 and IE9: page errors in editor, sometimes controls don't load, preventing editing - https://phabricator.wikimedia.org/T60464#1773959 (10zeljkofilipin) 5Open>3Invalid a:3zeljkofilipin We do not run tests in IE any... [16:11:40] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow: Flow browser test: hover failure in IE in flow_logged_in test - https://phabricator.wikimedia.org/T69723#1773982 (10SBisson) It can be closed as invalid since Flow browser tests in IE are not executed or supported for the time being. [16:12:01] fpfpaaa [16:12:03] salt never works [16:12:36] 10Browser-Tests, 10VisualEditor: VE cursor test invokes ULS IME in Firefox on test2wiki - https://phabricator.wikimedia.org/T57972#1773984 (10zeljkofilipin) Is this still a problem, or can this task be resolved? [16:13:03] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review, 3Wikidata-Sprint-2015-10-13: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1773987 (10Tobi_WMDE_SW) @janzerebecki I think that's blocked on T115500 [16:14:21] 10Browser-Tests, 6Commons, 6Multimedia, 10UploadWizard: UploadWizard API tests failing on beta Commons due to login problem - https://phabricator.wikimedia.org/T89272#1773991 (10zeljkofilipin) 5Open>3Resolved a:3zeljkofilipin The job is green now, resolving. Please reopen if needed. [16:15:12] !log integration-slave-trusty-1014 upgrading Zuul to /root/zuul_2.1.0-60-g1cc37f7-wmf1trusty1_amd64.deb [16:15:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:15:36] !log integration-slave-trusty-1014 upgrading Zuul to /root/zuul_2.1.0-60-g1cc37f7-wmf1trusty1_amd64.deb . In case of trouble unpool it from https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1014/ [16:15:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:15:53] 10Browser-Tests, 10MediaWiki-extensions-DonationInterface: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#1773996 (10zeljkofilipin) Is somebody planning to work on this, or can this task be resolved? [16:20:15] 10Browser-Tests: We should check for ResourceLoader errors after any page opens - https://phabricator.wikimedia.org/T59304#1774011 (10zeljkofilipin) Anybody still cares about this, or can this task be resolved? [16:20:48] 10Browser-Tests: browser tests should assert there are no pink error boxes on the page - https://phabricator.wikimedia.org/T63304#1774015 (10zeljkofilipin) Anybody still cares about this, or can this be resolved? [16:21:09] Hello, the new tech news issue was not delivered to dewiki, does anyone know why? [16:21:28] Luke081515: most probably not handled by us [16:22:30] hm, ok [16:22:44] Maybe a case for #wikimedia-tech? [16:22:55] 10Browser-Tests, 6Multimedia, 10UploadWizard: Links broken in UploadWizard-api-commons.wikimedia.beta.wmflabs.org job error report - https://phabricator.wikimedia.org/T66119#1774022 (10zeljkofilipin) Is this still a problem, or can this task be resolved? [16:23:41] 10Deployment-Systems, 3Scap3, 7Documentation: Regenerate scap documentation when using differential for code review - https://phabricator.wikimedia.org/T117431#1774024 (10demon) As I said in our last team meeting, we should probably just move this to being generated hourly or something instead of being trigg... [16:24:01] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow, 7Performance: QA: implement "login to " to speed up browser tests - https://phabricator.wikimedia.org/T73635#1774026 (10zeljkofilipin) 5Open>3Resolved a:3zeljkofilipin Browser tests can now log in via the API, making it really fast. Please r... [16:26:32] 10Browser-Tests, 7Documentation: write up "making sense of Jenkins browser test results" - https://phabricator.wikimedia.org/T87225#1774041 (10zeljkofilipin) Anybody plans to work on this, or can this task be resolved? [16:27:06] !log beta: restart salt-minion on a bunch of instances [16:27:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:27:48] 10Browser-Tests, 6Multimedia, 10UploadWizard: Run UploadWizard API tests after mediawiki/core - https://phabricator.wikimedia.org/T65105#1774044 (10zeljkofilipin) 5Open>3declined a:3zeljkofilipin I do not remember the context for this task any more. If you care about it, please reopen it and provide th... [16:30:48] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow, 7Browser-Support-Google-Chrome: upstream: Chrome sending bogus newline etc. values inside sendkeys strings - https://phabricator.wikimedia.org/T64536#1774058 (10zeljkofilipin) 5Open>3Invalid a:3zeljkofilipin No reply in a long time, resolving as i... [16:37:27] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review, 3Wikidata-Sprint-2015-10-13: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1774091 (10JanZerebecki) raita.wmflabs.org shows the correct link. [17:04:17] Could somebody help me when i try to go to https://git.wikimedia.org/commit/operations%2Fpuppet/9b350a1afa9944cb59923a42a546e9ef202381a6 i am getting Internal error. Not sure why. [17:18:35] 7Blocked-on-RelEng, 10Deployment-Systems, 3Scap3, 6Discovery: Create deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#1774338 (10mmodell) As far as I know this is ready to go on our end, I just need the go-ahead and I can start deploying this repo along with the train. [17:20:35] 7Blocked-on-RelEng, 10Deployment-Systems, 3Scap3, 6Discovery: Create deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#1703137 (10EBernhardson) added @MaxSem as a subscriber as he is now leading up the backend portion of the portals work. [17:27:35] interesting: Since yesterday, we have more than 50 abusefilter hits at beta, normaly we only have one a day [17:40:41] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 3Fundraising Sprint William Shatner, and 2 others: Tests on deployment branches of wikimedia/fundraising/crm falling causing to force merge (and deadlock of Zuul) - https://phabricator.wikimedia.org/T117062#1765798 (10awight)... [18:00:54] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Move `tmpdir` on deployment-db instances - https://phabricator.wikimedia.org/T117446#1774522 (10dduvall) 3NEW [18:04:14] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Move `tmpdir` on deployment-db instances - https://phabricator.wikimedia.org/T117446#1774544 (10dduvall) [18:05:42] 10Deployment-Systems, 3Scap3, 6Discovery: Create deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#1774555 (10greg) [18:09:27] 10Deployment-Systems, 3Scap3, 7Documentation: Regenerate scap documentation when using differential for code review - https://phabricator.wikimedia.org/T117431#1774580 (10mmodell) @demon: I tried to do that but I couldn't get the jenkins job to cooperate. [18:13:35] 5Continuous-Integration-Scaling, 6operations, 3labs-sprint-119: Allow network flow between labs instance and scandium - https://phabricator.wikimedia.org/T116975#1774620 (10chasemp) [18:17:26] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow, 15User-greg: Flow browser test: hover failure in IE in flow_logged_in test - https://phabricator.wikimedia.org/T69723#1774652 (10greg) 5Open>3Invalid a:3greg >>! In T69723#1773982, @SBisson wrote: > It can be closed as invalid since Flow browser t... [18:31:20] 10Continuous-Integration-Infrastructure, 10Deployment-Systems, 3Scap3, 7Documentation: Regenerate scap documentation when using differential for code review - https://phabricator.wikimedia.org/T117431#1774734 (10Legoktm) [18:33:04] 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1774753 (10greg) Just chatted with @santhosh in the office about this and another good wiki to potentially move would be Hebrew Wikipedia.... [18:40:18] 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10Flow, 3Collaboration-Team-Current: Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#1774775 (10Catrope) a:3Catrope [18:40:25] 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10Flow, 3Collaboration-Team-Current: Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#850716 (10Catrope) p:5Normal>3High [18:53:08] 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1774814 (10Amire80) I'll have to ask the community, but I like it. [18:54:39] 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1774816 (10Amire80) Is it Catalan **and** Hebrew? cawiki already agrees. [18:57:22] 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1774824 (10greg) @Amire80 As long as that's ok with everyon, yes :) Sorry, @Santhosh and I were just talking about Hebrew in the office. [19:29:26] 6Release-Engineering-Team, 3Scap3, 10Wikimedia-Developer-Summit-2016: Scap3: updates, upgrades, and challenges - https://phabricator.wikimedia.org/T114045#1774967 (10dduvall) Scap3 was developed chiefly to replace (and improve upon) Trebuchet as a deployment tool for Services but with a general enough archit... [19:35:49] greg-g: can I deploy UrlShortener to beta cluster? And do I need a window for it or can I do it whenever? [19:37:48] 6Release-Engineering-Team, 3Scap3, 10Wikimedia-Developer-Summit-2016: Scap3: updates, upgrades, and challenges - https://phabricator.wikimedia.org/T114045#1775028 (10Krenair) I am interested in this. [19:41:25] Am I just murdering Zuul with the forced submits? [19:41:47] ah, it's going now... [19:50:45] (03PS1) 10Legoktm: Make the wikimedia-fundraising-cvicrm regex more aggressive?? [integration/config] - 10https://gerrit.wikimedia.org/r/250487 [19:50:54] awight: ^ sanity check? [19:50:59] no idea whether it'll work [19:51:44] legoktm: yes and whenever (I assume it's been security reviewed?) [19:51:53] yep [19:51:59] * greg-g nods [19:52:22] legoktm: nothing to lose but our chains! [19:52:44] (03CR) 10Awight: [C: 031] "Desperate times! Thanks for looking at this." [integration/config] - 10https://gerrit.wikimedia.org/r/250487 (owner: 10Legoktm) [19:52:58] (03CR) 10Legoktm: [C: 032] Make the wikimedia-fundraising-cvicrm regex more aggressive?? [integration/config] - 10https://gerrit.wikimedia.org/r/250487 (owner: 10Legoktm) [19:54:05] (03PS2) 10Awight: Make the wikimedia-fundraising-cvicrm regex more aggressive?? [integration/config] - 10https://gerrit.wikimedia.org/r/250487 (https://phabricator.wikimedia.org/T117062) (owner: 10Legoktm) [19:54:31] (03CR) 10Legoktm: Make the wikimedia-fundraising-cvicrm regex more aggressive?? [integration/config] - 10https://gerrit.wikimedia.org/r/250487 (https://phabricator.wikimedia.org/T117062) (owner: 10Legoktm) [19:54:31] (added the Bug#, is all) [19:54:35] (03CR) 10Legoktm: [C: 032] Make the wikimedia-fundraising-cvicrm regex more aggressive?? [integration/config] - 10https://gerrit.wikimedia.org/r/250487 (https://phabricator.wikimedia.org/T117062) (owner: 10Legoktm) [19:59:58] (03Merged) 10jenkins-bot: Make the wikimedia-fundraising-cvicrm regex more aggressive?? [integration/config] - 10https://gerrit.wikimedia.org/r/250487 (https://phabricator.wikimedia.org/T117062) (owner: 10Legoktm) [20:00:10] !log deploying https://gerrit.wikimedia.org/r/250487 [20:00:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:00:55] awight: deployed [20:02:39] legoktm: rad, thanks! I'll holler when I can confirm or deny that it works [20:06:02] 10Continuous-Integration-Infrastructure, 10Deployment-Systems, 3Scap3, 7Documentation: Regenerate scap documentation when using differential for code review - https://phabricator.wikimedia.org/T117431#1775149 (10hashar) In an ideal world, in the Zuul layout one would just: ``` lang=diff - name: mediawik... [20:12:34] 10Beta-Cluster-Infrastructure, 7Database: Investigate slow query logging/digest for Beta Cluster - https://phabricator.wikimedia.org/T116793#1775166 (10hashar) p:5Triage>3Normal Will poke @jcrespo about it, since we are in the same timezone it is more convenient. [20:18:09] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Move `tmpdir` on deployment-db instances - https://phabricator.wikimedia.org/T117446#1775177 (10hashar) On the labs instances we are using a puppet class that allocates all the extended disk to a single partition /mnt with lvm. End result for db2: | Fi... [20:22:44] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [20:27:57] thcipriani: can staging-test-tin survive a few minutes offline? [20:28:15] andrewbogott: sure, no problem. staging is not doing much. [20:28:22] great, thanks [20:30:58] 10Beta-Cluster-Infrastructure: beta cluster missing vips command needed to render tiffs and pngs, pnmtojpeg for DjVus - https://phabricator.wikimedia.org/T116816#1775253 (10thcipriani) p:5Triage>3Normal [20:31:28] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Move `tmpdir` on deployment-db instances - https://phabricator.wikimedia.org/T117446#1775258 (10thcipriani) p:5Triage>3Normal [20:32:01] RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms [20:34:22] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-fluorine fails puppet '/usr/sbin/usermod -u 10003 datasets' returned 4: usermod: UID '10003' already exists - https://phabricator.wikimedia.org/T117028#1775278 (10hashar) p:5Triage>3Normal [20:35:33] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Database, 7Database, 7WorkType-NewFunctionality: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255#1775284 (10hashar) p:5Triage>3Normal [20:35:53] 10Beta-Cluster-Infrastructure, 6Analytics-Engineering, 5Patch-For-Review, 7Varnish, 7WorkType-Maintenance: On beta cluster varnish stats process points to production statsd - https://phabricator.wikimedia.org/T116898#1775285 (10hashar) p:5Triage>3Normal [20:41:14] 10Continuous-Integration-Infrastructure, 10Deployment-Systems, 3Scap3, 7Documentation: Regenerate scap documentation when using differential for code review - https://phabricator.wikimedia.org/T117431#1775312 (10hashar) See also https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation [20:44:52] !sal [20:44:52] https://tools.wmflabs.org/sal/releng [20:48:10] !log Upgraded Zuul on Trusty slaves to /root/zuul_2.1.0-60-g1cc37f7-wmf1trusty1_amd64.deb [20:48:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:59:16] 10Continuous-Integration-Infrastructure: Find out which jobs are using ElasticSearch as a backend - https://phabricator.wikimedia.org/T112667#1775361 (10hashar) a:5hashar>3None [21:07:32] 5Continuous-Integration-Scaling, 6operations, 5Patch-For-Review: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1775415 (10chasemp) [21:07:33] 5Continuous-Integration-Scaling, 6operations, 3labs-sprint-119: Allow network flow between labs instance and scandium - https://phabricator.wikimedia.org/T116975#1775413 (10chasemp) 5Open>3Resolved this should all work now [21:07:51] 10Beta-Cluster-Infrastructure: Set up graphite monitoring for the beta cluster - https://phabricator.wikimedia.org/T54357#1775419 (10hashar) [21:07:52] 10Beta-Cluster-Infrastructure, 5Patch-For-Review: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#1775416 (10hashar) 5Open>3Resolved a:3hashar This has been done as part of the Monolog refactoring and we have: ``` wmf-config/CommonSettings.php:... [21:08:26] 10Beta-Cluster-Infrastructure: Setup Gdash for beta cluster - https://phabricator.wikimedia.org/T75882#1775427 (10hashar) [21:08:27] 10Beta-Cluster-Infrastructure, 5Patch-For-Review: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#1775426 (10hashar) [21:08:50] 10Beta-Cluster-Infrastructure, 5Patch-For-Review: Send MediaWiki profiling on beta cluster to graphite.wmflabs.org - https://phabricator.wikimedia.org/T75881#784292 (10hashar) [21:08:52] 10Beta-Cluster-Infrastructure: Setup Gdash for beta cluster - https://phabricator.wikimedia.org/T75882#1775438 (10hashar) [21:10:28] Yippee, build fixed! [21:10:28] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #816: 09FIXED in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/816/ [21:11:10] 10Beta-Cluster-Infrastructure: Setup Grafana for beta cluster - https://phabricator.wikimedia.org/T75882#1775443 (10hashar) [21:11:29] 10Beta-Cluster-Infrastructure: Setup Grafana for beta cluster - https://phabricator.wikimedia.org/T75882#784303 (10hashar) gdash is gone. So I have repurposed this task to use Grafana instead. [21:16:10] 10Beta-Cluster-Infrastructure: Upgrade varnish automatically via puppet in Beta Cluster - https://phabricator.wikimedia.org/T75564#1775454 (10hashar) All beta cluster varnish caches have been upgraded to Jessie (T98758 and T103660). The Debian `unattended-upgrades` system is apparently enabled, so potentially n... [21:18:24] 10Beta-Cluster-Infrastructure, 6operations, 5Patch-For-Review: mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - https://phabricator.wikimedia.org/T67591#1775460 (10hashar) [21:18:25] 10Beta-Cluster-Infrastructure, 6operations, 5Patch-For-Review: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#1775458 (10hashar) 5Open>3Resolved Seems this one has been fixed ages ago. [21:23:38] 10Beta-Cluster-Infrastructure: Properly rotate udp2log logs (restart daemon) - https://phabricator.wikimedia.org/T72112#1775474 (10hashar) 5Open>3Resolved a:3hashar udp2log is gone since May 11th. We are now using Monolog / logstash. [21:28:45] 10Beta-Cluster-Infrastructure: sync Sandbox gadget from production to en.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T49205#1775494 (10hashar) [21:28:47] 10Beta-Cluster-Infrastructure, 7JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779#1775495 (10hashar) [21:29:22] 10Beta-Cluster-Infrastructure: sync-site-resources should sync all Labs wikis - https://phabricator.wikimedia.org/T51791#1775505 (10hashar) [21:29:23] 10Beta-Cluster-Infrastructure, 7JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779#1775506 (10hashar) [21:31:23] 10Beta-Cluster-Infrastructure, 7Tracking: automatically import some content from production into Beta-Cluster (tracking) - https://phabricator.wikimedia.org/T54382#1775516 (10hashar) Lets use T51779 [21:31:38] 10Beta-Cluster-Infrastructure: sync-site-resources should sync all Labs wikis - https://phabricator.wikimedia.org/T51791#1775520 (10hashar) [21:31:39] 10Beta-Cluster-Infrastructure, 7JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779#1775521 (10hashar) [21:31:40] 10Beta-Cluster-Infrastructure, 7Tracking: automatically import some content from production into Beta-Cluster (tracking) - https://phabricator.wikimedia.org/T54382#1775519 (10hashar) [21:31:50] 10Beta-Cluster-Infrastructure, 7Tracking: automatically import some content from production into Beta-Cluster (tracking) - https://phabricator.wikimedia.org/T54382#590234 (10hashar) [21:31:51] 10Beta-Cluster-Infrastructure, 7JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779#576690 (10hashar) [21:32:07] 10Beta-Cluster-Infrastructure: sync Sandbox gadget from production to en.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T49205#1775524 (10hashar) [21:32:09] 10Beta-Cluster-Infrastructure, 7JavaScript: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779#576690 (10hashar) [21:32:25] 10Beta-Cluster-Infrastructure: sync Sandbox gadget from production to en.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T49205#522721 (10hashar) [21:39:07] chasemp: thanks for the network/firewall flows :-} [21:39:22] sure thing [22:04:17] can someone help me get access to mysql on deployment-eventlogging03.eqiad.wmflabs [22:04:52] jdlrobson: #wikimedia-analytics might help [22:04:59] lol [22:05:02] jdlrobson: I think they maintain the whole event stack :D [22:05:04] gone full circle [22:05:08] :-( [22:05:08] ha I told him to ask here maybe that's my bad [22:05:24] #wikimedia-analytics sent me to #wikimedia-labs which sent me here [22:05:27] is deployment-eventlogging03.eqiad.wmflabs on betacluster? [22:05:33] yup [22:06:04] maybe the password is in labs/private.git as a cherry pick on deployment-puppetmaster [22:07:04] role::eventlogging, role::eventlogging::consumer::files, role::eventlogging::consumer::mysql, role::eventlogging::forwarder, role::eventlogging::processor, role::labs::lvm::srv, role::labs::instance, role::puppet::self [22:07:05] bah [22:07:48] hashar: ah I see so analytics manages mysql there on their own jdlrobson sorry I steered you wrong [22:07:53] class { 'passwords::mysql::eventlogging': } # T82265 [22:07:59] $mysql_user = $passwords::mysql::eventlogging::user [22:07:59] $mysql_pass = $passwords::mysql::eventlogging::password [22:07:59] $mysql_db = $::realm ? { [22:07:59] production => 'm4-master.eqiad.wmnet/log', [22:08:01] labs => '127.0.0.1/log', [22:08:01] } [22:08:08] so luckily beta and prod are alike [22:08:20] and passwords::mysql::eventlogging::password should be either in labs private [22:08:23] yeah thats probably pulling from the labs "private" right? [22:08:24] yeah [22:08:42] or as a cherry pick on deployment-puppetmaster in /var/lib/git/labs/private [22:08:48] jdlrobson: look in https://gerrit.wikimedia.org/r/#/admin/projects/labs/private [22:08:49] jdlrobson: ^^^^ wall of text for you :-} [22:09:17] boom! [22:09:17] in [22:09:23] thanks guys~ [22:09:33] * hashar whistles [22:09:53] I have no clue whether it is documented somewhere [22:10:03] kafka/ eventlogging is a mystery to me :} [22:12:19] * hashar waves [22:20:11] 10Beta-Cluster-Infrastructure, 6operations, 5Patch-For-Review: mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - https://phabricator.wikimedia.org/T67591#1775741 (10yuvipanda) [22:20:12] 10Beta-Cluster-Infrastructure, 6operations, 5Patch-For-Review: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#1775739 (10yuvipanda) 5Resolved>3Open No I just reopened it see T86668#1704524 [22:20:38] (03CR) 10Chad: "Is this still needed considering https://phabricator.wikimedia.org/D16?" [tools/scap] - 10https://gerrit.wikimedia.org/r/241066 (https://phabricator.wikimedia.org/T113073) (owner: 10Thcipriani) [22:21:38] Project beta-update-databases-eqiad build #4100: 04FAILURE in 1 min 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/4100/ [22:22:29] (03Abandoned) 10Thcipriani: [WIP] Add optional canary deploy group and check [tools/scap] - 10https://gerrit.wikimedia.org/r/241066 (https://phabricator.wikimedia.org/T113073) (owner: 10Thcipriani) [22:23:44] thcipriani: thx [22:23:52] ostriches: nice catch :) [22:24:40] I'd /like/ to cut Gerrit out of the picture for scap so I'm trying to close the last 2 patches :) [22:24:47] so besides landing ostriches, differential > gerrit? [22:25:12] Pretty much ya [22:25:19] "landing ostriches" == playing Joust? [22:25:41] bd808: Angry Ostriches, my new mobile app. [22:26:19] I smell a lawsuit [22:26:21] :) [22:26:22] fighting the evil devs who make crufty code [22:26:44] http://www.kevincharnas.com/uploaded_images/angry_ostrich-780215.jpg [22:26:57] that is messed up [22:27:08] (03Draft1) 10BryanDavis: Find merge-base for git repos with multiple remotes [tools/scap] - 10https://gerrit.wikimedia.org/r/185104 (https://phabricator.wikimedia.org/T86811) [22:27:15] PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145) [22:27:23] bd808: Undrafted so we can decide if we land or abandon [22:27:40] easy choice [22:28:05] (03Abandoned) 10BryanDavis: Find merge-base for git repos with multiple remotes [tools/scap] - 10https://gerrit.wikimedia.org/r/185104 (https://phabricator.wikimedia.org/T86811) (owner: 10BryanDavis) [22:28:39] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Unplanned-Sprint-Work, 3Fundraising Sprint William Shatner, and 2 others: Tests on deployment branches of wikimedia/fundraising/crm falling causing to force merge (and deadlock of Zuul) - https://phabricator.wikimedia.org/T117062#1775802 (10awight)... [22:29:01] No open patches in scap :D [22:29:24] 10Deployment-Systems, 5Patch-For-Review: scap barfs if an extension has more than one remote - https://phabricator.wikimedia.org/T86811#1775805 (10bd808) 5Open>3Resolved a:3bd808 >>! In T86811#977614, @Reedy wrote: > Indeed. I fixed it by just removing the gerrit remote... [22:29:44] 10Deployment-Systems, 5Patch-For-Review: scap barfs if an extension has more than one remote - https://phabricator.wikimedia.org/T86811#1775808 (10bd808) a:5bd808>3Reedy [22:29:45] I do envy your post gerrit world heavily [22:31:26] chasemp: If we fetched scap (the copy on tin/deployment-bastion) from Phab, I could mark the repo as readonly [22:31:45] do you plan to do that? [22:33:16] I can write up a patch easy enough [22:33:19] it should be a simple puppet change [22:33:43] Does it adjust the existing remote or will that need manual cleanup? [22:33:48] it would only effect the deploy masters (deployment-bastion and tin) [22:34:05] Yeah [22:34:07] It would need manual clean up I'm pretty sure [22:34:12] I thought so, nbd [22:34:16] neat [22:34:26] bd808: on https://phabricator.wikimedia.org/T113571 is it sane to have a short lived retention period of 24h for some verbose but useful logs? [22:34:46] well.... [22:34:50] i.e. route verbose things to some smaller time period rollover [22:34:51] possibly [22:35:06] we did that at my last gig I believe and it worked but I have no idea how much trouble [22:35:09] but to do that we would really want to add separate indexes for them [22:35:20] yeah [22:35:29] a "delete" operation in elastic is pretty gross [22:35:30] https://gerrit.wikimedia.org/r/#/c/250578/ up for review [22:35:37] I'm not sure runJobs logs is a good candidate anyways [22:35:40] but something will be [22:35:58] We have the tech to do it actually [22:36:14] I think we used it for file uploads and such logs [22:36:27] as if it's longer than 72 hours there asnt a chance of hell in debugging [22:36:32] I use it the opposite way for the sal and bash tools (index that only rotates once a month) [22:36:35] but within the range you wanted all teh details you can get etc [22:37:14] anyway, not advocating just floating the idea [22:37:19] *nod* [22:37:59] If we had really high volume stuff I think I'd want to create hourly indexes to hold them and kill those off on a regular basis [22:38:25] oen thing you have to look out for in elsaticsearch is putting too many documents into a shard [22:38:57] we are already pushing the limits of good taste there with our one per day logstash indexes actually [22:39:14] at our volume I imagine so [22:39:57] <80M events per day [22:40:18] actually <100M most days [22:40:44] yeah once you start directly battling lucene internals you've lost [22:41:19] s// [22:42:36] "< bd808> "landing ostriches" == playing Joust?" OMG [22:42:55] * bd808 is old [22:43:15] hey, I played that in the arcade all the time! [22:45:29] Joist strategy is a major plot point in "ready player one" [22:46:56] didn't see it [22:47:39] you made me thikn of this so here tho it's unrelated to anything https://www.youtube.com/watch?v=1BvFcrna6aI [22:48:58] greg-g: it's a book with words and stuff :) -- https://en.wikipedia.org/wiki/Ready_Player_One [23:07:01] fwiw, i interviewed an ops candidate last week that admins 14 elasticsearch clusters than have an aggregate multi-PB of storage for logging, his answer to how to do that was buy more hardware :P [23:07:21] (well, more detailed but that was the short of it :) [23:08:11] We kind of knew that when we expanded last time, we certainly didn't buy enough for long term growth [23:20:50] uh, how long does beta scap normally take? https://integration.wikimedia.org/ci/job/beta-scap-eqiad/77038/console [23:20:58] it's been 12 minutes... [23:21:33] That stage should only take a minute, two tops. [23:21:40] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/buildTimeTrend I see a 40 minutes run :/ [23:21:46] " Updating LocalisationCache" can take a long time but I thought it sprew a list of languages being updated [23:21:54] this scap is pulling in a new extension (UrlShortener) [23:22:08] it will be slow [23:22:29] l10nupdate stuff on deployment-bastion is horribly slow [23:22:40] Yippee, build fixed! [23:22:40] Project beta-update-databases-eqiad build #4101: 09FIXED in 2 min 38 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/4101/ [23:22:41] that vm is really underpowered [23:22:49] It does too much. [23:22:57] that too [23:22:58] We need to split bastion / deploy master. [23:23:14] hmm [23:23:20] https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/4101/console says "23:22:39 Creating urlshortcodes table ...done." [23:23:37] oh, that's my bad [23:24:19] * legoktm waits patiently [23:24:55] looks like the --verbose flag got yanked from the job's shell script :( [23:25:17] I think if it was set to verbose you'd see more going on in the log [23:29:52] dr0ptp4kt: https://phabricator.wikimedia.org/T78186 [23:34:05] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Rebuild deployment master - https://phabricator.wikimedia.org/T117504#1775986 (10demon) 3NEW a:3demon [23:39:45] 23:08:29 23:08:29 Updating LocalisationCache for master using 2 thread(s) [23:39:45] 23:36:32 23:36:32 Generating JSON versions and md5 files [23:39:52] it's syncing now :D