[00:07:50] Krenair: http://w-beta.wmflabs.org/ isn't redirecting to http://meta.wikimedia.beta.wmflabs.org/wiki/Special:UrlShortener :| [00:07:53] 5Release-Engineering-Epics, 10Gather, 10MobileFrontend, 10Reading Web Planning, 7Epic: [EPIC] Create a formal release process for MobileFrontend/Gather - https://phabricator.wikimedia.org/T100296#1776149 (10Jdlrobson) @phuedx @jhernandez this is done right? [00:09:29] w-beta.wmflabs.org/2 works though :D [00:37:07] legoktm, http://serverfault.com/questions/605931/can-you-use-redirect-and-proxypass-at-the-same-time might be helpful [00:38:05] thanks, I'll take a look in a bit [00:53:08] 10Deployment-Systems, 3Scap3, 6Discovery: Create deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#1776322 (10MaxSem) p:5High>3Low Lowering priority: with https://gerrit.wikimedia.org/r/#/c/248526/ , we're fine for now. Having this repository deployed via scap3 (and potentially... [00:55:46] 10Deployment-Systems, 3Scap3, 6Discovery: Create scripts for automatic deployment for wikimedia/portals - https://phabricator.wikimedia.org/T114694#1776336 (10ksmith) [02:13:56] Did something change about the beta-scap-eqiad job? It seems to be taking > 30 minutes more often than before (with the rest of the runs taking a minute or two like of old), but maybe I'm just imagining it. [02:17:11] bd808, ^ [02:18:22] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/77047/console [02:18:39] 01:35:34 01:35:34 Updating LocalisationCache for master using 2 thread(s) [02:18:39] 02:03:42 02:03:42 Generating JSON versions and md5 files [02:18:39] 02:05:51 02:05:51 Finished mw-update-l10n (duration: 30m 18s) [02:21:57] greg-g, James_F: So none of the ldap admins are watching https://phabricator.wikimedia.org/tag/ldap-access-requests/ ? [02:22:49] Krenair: Seemingly not. :-( [04:18:27] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<33.33%) [06:38:27] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [08:32:59] Yippee, build fixed! [08:32:59] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #771: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/771/ [09:10:44] 10Browser-Tests, 10Wikidata: No test report files were found: job fails in jenkins but is shown as successful in raita - https://phabricator.wikimedia.org/T116164#1776705 (10adrianheine) When executing the cucumber command locally, the result directory is filled correctly. [Last successful run](https://integra... [09:30:37] PROBLEM - Puppet failure on deployment-fluorine is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:37:10] 10Browser-Tests, 10Wikidata: No test report files were found: job fails in jenkins but is shown as successful in raita - https://phabricator.wikimedia.org/T116164#1776741 (10hashar) From an IRC discussion on 2015-10-27 in #wikidata: ``` lang=irc hashar: Can you help us/me fixing https://integrati... [09:43:43] 10Browser-Tests, 10Wikidata: No test report files were found: job fails in jenkins but is shown as successful in raita - https://phabricator.wikimedia.org/T116164#1776747 (10hashar) I ran the job manually directly on the slave reusing the env variables and I apparently I can not reproduce :-( [09:47:15] 10Beta-Cluster-Infrastructure, 6operations: [OPS] udp2log prevents udp2log-mw from starting - https://phabricator.wikimedia.org/T40995#1776756 (10hashar) 5Open>3declined a:3hashar udp2log is gone, at least from bastion. [09:49:40] 10Beta-Cluster-Infrastructure: Enable image rotation on beta for testing purposes - https://phabricator.wikimedia.org/T105877#1776759 (10hashar) 5Open>3declined a:3hashar I am poking a message on T105877 to hint about enabling the image rotation API on beta cluster. Closing this task since there is appare... [09:55:33] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Rebuild deployment master - https://phabricator.wikimedia.org/T117504#1776775 (10hashar) By deployment master are you referring to `deployment-bastion` or `mira.deployment-prep.eqiad.wmflabs`? The later has: * puppet class `role::deployment::server` *... [09:57:25] PROBLEM - Host deployment-cache-parsoid04 is DOWN: CRITICAL - Host Unreachable (10.68.19.197) [10:19:55] 10Browser-Tests, 10Wikidata: No test report files were found: job fails in jenkins but is shown as successful in raita - https://phabricator.wikimedia.org/T116164#1776804 (10adrianheine) I suppose you mean you cannot reproduce the issue, right? Fyi, it's indeed only one scenario that should be run, so that's a... [10:51:25] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Scaling, 5Patch-For-Review: Nodepool images need Gerrit mirror for git-clone performance - https://phabricator.wikimedia.org/T87294#1776864 (10hashar) [11:03:26] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-11-03: [Bug] the changed job configuration extension-unittests -> extension-unittests-generic for Wikidata.git makes it not run all tests and fail - https://phabricator.wikimedia.org/T95897#1776876 (10JanZerebecki) [11:03:37] 10Continuous-Integration-Config, 10Wikidata, 3Wikidata-Sprint-2015-11-03: [Task] Add Wikidata to Jenkins job mediawiki-extensions-hhvm - https://phabricator.wikimedia.org/T96264#1776877 (10JanZerebecki) [11:17:27] (03CR) 10Hashar: [C: 031] "At first glance that looks fine and the Jenkins credential store has a WikidataTester user" [integration/config] - 10https://gerrit.wikimedia.org/r/247901 (https://phabricator.wikimedia.org/T116166) (owner: 10JanZerebecki) [11:22:23] (03PS3) 10Hashar: [GoogleLogin] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/250195 (owner: 10Paladox) [11:24:43] (03CR) 10Hashar: [C: 032] [GoogleLogin] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/250195 (owner: 10Paladox) [11:25:57] (03Merged) 10jenkins-bot: [GoogleLogin] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/250195 (owner: 10Paladox) [12:15:29] (03PS8) 10Hashar: Create 'puppet-doc' macro and use it [integration/config] - 10https://gerrit.wikimedia.org/r/204983 (owner: 10Legoktm) [12:26:53] (03CR) 10Hashar: [C: 04-1] "We changed the operations-puppet-doc templates between rebased of this change. It now fetch code under $WORKSPACE/src unlike the vagrant j" [integration/config] - 10https://gerrit.wikimedia.org/r/204983 (owner: 10Legoktm) [12:34:39] (03PS1) 10Phedenskog: Test the desktop site at WPT.org [integration/config] - 10https://gerrit.wikimedia.org/r/250672 [12:41:32] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review, and 2 others: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1777045 (10Tobi_WMDE_SW) [12:48:50] 10Deployment-Systems, 6Release-Engineering-Team: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1777073 (10Amire80) I proposed this in the [[ https://he.wikipedia.org/wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%9E%D7%96%D7... [13:05:33] 10Browser-Tests, 10MediaWiki-extensions-WikibaseView, 10Wikidata: [Task] Browsertests for focus flow - https://phabricator.wikimedia.org/T50136#1777085 (10adrianheine) [13:07:06] !log Upgrading Jenkins plugin "Green Ball" from 1.14 to 1.15. Seems to fix a potential deadlock on jenkins start ( https://issues.jenkins-ci.org/browse/JENKINS-28422 ) [13:07:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:09:25] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #837: 04FAILURE in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/837/ [13:09:47] !log Jenkins upgrading a few more plugins [13:09:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:20:30] !log restarting Jenkins to apply updated plugins [13:20:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:24:22] (03CR) 10Hashar: [C: 031] Test the desktop site at WPT.org [integration/config] - 10https://gerrit.wikimedia.org/r/250672 (owner: 10Phedenskog) [14:13:34] 10Beta-Cluster-Infrastructure, 6Labs, 10Labs-Infrastructure, 7Graphite, and 2 others: Delete more specific deployment-prep graphite datapoints - https://phabricator.wikimedia.org/T111540#1777190 (10fgiunchedi) @krenair I've ran `archive-instances` on `labmon1001` so the deployment-prep hosts are gone, not... [14:27:27] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Rebuild deployment master - https://phabricator.wikimedia.org/T117504#1777208 (10demon) `mira` was just a testing box. `deployment-bastion` will remain as a bastion. This is about a `deployment-tin` (that I started yesterday) that will be an xlarge. [15:32:00] 10Beta-Cluster-Infrastructure, 6Labs, 10Labs-Infrastructure, 7Graphite, 7Shinken: Delete more specific deployment-prep graphite datapoints - https://phabricator.wikimedia.org/T111540#1777412 (10Krenair) [15:32:34] 10Beta-Cluster-Infrastructure, 6Labs, 10Labs-Infrastructure, 7Graphite, 7Shinken: Delete more specific deployment-prep graphite datapoints - https://phabricator.wikimedia.org/T111540#1606894 (10Krenair) How do we detect that those exist in the first place? [15:33:26] 10Differential, 5Gerrit-Migration, 10Security-Reviews: security review of phabricator.w.o before being used for git hosting and code review - https://phabricator.wikimedia.org/T117552#1777427 (10JanZerebecki) 3NEW [15:55:11] ostriches: we should probably coordinate https://gerrit.wikimedia.org/r/#/c/250578/ then, let me know when you want me to merge and you can do the cleanup post? [15:56:07] PROBLEM - Puppet failure on integration-slave-trusty-1017 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [15:56:41] chasemp: This afternoon work? Got a doc appt this morning. [15:56:49] ostriches: sure [16:27:36] 10Browser-Tests: selenium fails to connect to firefox (headless not sauce) - https://phabricator.wikimedia.org/T117561#1777674 (10JanZerebecki) 3NEW [16:28:10] 10Browser-Tests: selenium fails to connect to firefox (headless not sauce) - https://phabricator.wikimedia.org/T117561#1777683 (10JanZerebecki) To make this work: try to do the same as the gerrit triggered build, see mw-set-env-mw-selenium.sh in jenkins repo. [16:29:19] 5Release-Engineering-Epics, 10Gather, 10MobileFrontend, 10Reading Web Planning, 7Epic: [EPIC] Create a formal release process for MobileFrontend/Gather - https://phabricator.wikimedia.org/T100296#1777686 (10phuedx) @Jdlrobson: Yarrrp. [16:32:27] 10Browser-Tests: browsertest failure reports don't show the failing tests saucelabs link, but a different one - https://phabricator.wikimedia.org/T115500#1777705 (10JanZerebecki) This might have been fixed in wikimedia-selenium 1.6.2. [16:33:40] RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.87 ms [16:35:10] 10Browser-Tests, 10Wikidata: upgrade wikidata browser tests to wikimedia-selenium 1.6.2 - https://phabricator.wikimedia.org/T117562#1777716 (10JanZerebecki) 3NEW [16:36:07] RECOVERY - Puppet failure on integration-slave-trusty-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [16:37:14] 10Browser-Tests, 10Wikidata: upgrade wikidata browser tests to wikimedia-selenium 1.6.2 - https://phabricator.wikimedia.org/T117562#1777716 (10JanZerebecki) [16:39:43] 10Browser-Tests, 10Wikidata: upgrade wikidata browser tests to wikimedia-selenium 1.6.2 - https://phabricator.wikimedia.org/T117562#1777748 (10JanZerebecki) p:5Triage>3Normal [16:40:13] 10Browser-Tests, 10Wikidata, 3Wikidata-Sprint-2015-10-13: upgrade wikidata browser tests to wikimedia-selenium 1.6.2 - https://phabricator.wikimedia.org/T117562#1777716 (10JanZerebecki) [16:41:58] 10Browser-Tests, 10Wikidata, 3Wikidata-Sprint-2015-11-03: upgrade wikidata browser tests to wikimedia-selenium 1.6.2 - https://phabricator.wikimedia.org/T117562#1777755 (10JanZerebecki) [16:51:30] 10Browser-Tests, 10Wikidata, 3Wikidata-Sprint-2015-11-03: AuthorityControl gadget browsertest fail - https://phabricator.wikimedia.org/T117564#1777775 (10JanZerebecki) 3NEW [16:52:20] 10Browser-Tests, 10Wikidata, 7Regression, 3Wikidata-Sprint-2015-11-03: AuthorityControl gadget browsertest fail - https://phabricator.wikimedia.org/T117564#1777775 (10JanZerebecki) [16:59:38] Is there no meeting today, or did I just accidentally delete it from my calendar? [17:00:48] 10Browser-Tests, 10Wikidata, 7Regression, 3Wikidata-Sprint-2015-11-03: AuthorityControl gadget browsertest fail - https://phabricator.wikimedia.org/T117564#1777828 (10aude) I have updated the authority control gadget on beta. I was getting js errors and now it works for me. Maybe it also works again for... [17:01:07] PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145) [17:02:27] * andrewbogott ’s week is suspiciously meeting-free [17:03:34] andrewbogott: probably "meetingless Andrew week" [17:03:50] greg-g: what or who creates new weekly entries in the deployment calendar? [17:04:09] andrewbogott: me, sadly [17:04:16] copy/pasta [17:04:18] :( [17:04:40] greg-g: great. So, my goal is to not be added under puppet swat every damn week. Which entry do you c/p? I’ll just remove myself from that one. [17:05:01] andrewbogott: the the latest week on page [17:05:03] (step two is harder, which is training ops to add themselves as appropriate) [17:05:11] yup :) [17:05:12] by ‘latest’ you mean ‘bottom’? [17:05:42] yeah, furthest in the time space continuum [17:06:38] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Rebuild deployment master - https://phabricator.wikimedia.org/T117504#1777854 (10Luke081515) [17:07:16] greg-g: ok, got it, thanks. [17:07:41] greg-g: if you think of it, when you c/p you should probably blank out the puppet swat entries so there’s no the illusion of someone doing it when no on has volunteered. [17:07:58] andrewbogott: then it won't be scheduled :) [17:08:03] since it rotates every week, unlike normal swat [17:08:04] what i mean is, someone has to own it [17:08:07] ah [17:08:11] is that documented somewhere? [17:08:27] or just decided in the ops meeting on monday? [17:08:34] well, hm. We do always make sure someone is designated in the weekly meeting. It’s just a question of getting it on the calendar. [17:08:51] it’s usually planned in advance a bit. But we’re not great at having records outside of our meeting notes. [17:09:12] so, e.g. https://office.wikimedia.org/wiki/Operations/Operations_Meeting_Notes/TechOps-2015-10-26 [17:09:56] * greg-g nods [17:10:11] if I can count on ya'll to add the name on the deploy calendar page, I can blank it out during my copy/pasta [17:10:35] greg-g: I’m sending an email, will cc: you [17:11:22] kk [17:19:14] * andrewbogott sends a “who will do this?” email, which is the moral opposite of just doing it. [17:20:16] could the joust thign just say "ops" instead of a person [17:20:35] and we handle teh rotation internally and queries about who go to clinic person? [17:20:38] chasemp: it could, but part of the service we provide is telling people who to bug about it. [17:20:41] I have no idea how far outside of the norm taht is [17:21:06] I’m leaning towards thinking that clinic duty should include communicating who is in charge of what this week, yes. [17:21:09] no strong opinion sure but we could punt to clinic duty person in case of doubt [17:21:12] yeah [17:26:22] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira / deployment-bastion permissions problem - https://phabricator.wikimedia.org/T117016#1777944 (10Krenair) ```krenair@tin:~$ ls -al /srv | grep mediawiki-staging drwxrwsr-x 28 mwdeploy wikidev... [17:26:50] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow: Flow: topic created by logged in user was stored as created by anon user. - https://phabricator.wikimedia.org/T75926#1777947 (10SBisson) 5Open>3Invalid a:3SBisson Has not been seen in a long time. Please reopen if it happens again. [17:29:17] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow: Flow: logic of edit_existing browser test is broken - https://phabricator.wikimedia.org/T66082#1777966 (10SBisson) 5Open>3Invalid a:3SBisson Not relevant anymore with the current state of those tests. [17:34:24] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow: Flow firefox-monobook-sauce browser test has failed since Dec 5 in Reply moderation.Hiding a comment - https://phabricator.wikimedia.org/T85497#1778001 (10SBisson) 5Open>3Invalid a:3SBisson I don't know what it was but reply_moderation.feature:10 (h... [17:34:44] 10Browser-Tests, 10Wikidata, 3Wikidata-Sprint-2015-11-03: No test report files were found: job fails in jenkins but is shown as successful in raita - https://phabricator.wikimedia.org/T116164#1778004 (10JanZerebecki) [17:36:11] (03PS4) 10JanZerebecki: Additionally run Wikidata browsertests without saucelabs [integration/config] - 10https://gerrit.wikimedia.org/r/247901 (https://phabricator.wikimedia.org/T116166) [17:36:18] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow: Flow QA: Lock post action is offscreen and browser test fails - https://phabricator.wikimedia.org/T72878#1778012 (10SBisson) 5Open>3Invalid a:3SBisson Not relevant anymore with the current state of those tests. [17:37:16] 10Differential, 5Gerrit-Migration, 10Security-Reviews: security review of phabricator.w.o before being used for git hosting and code review - https://phabricator.wikimedia.org/T117552#1778018 (10mmodell) The phabricator upstream takes security pretty seriously. [17:41:43] 10Browser-Tests, 5Release-Engineering-Epics, 7Epic, 7Tracking: Fix or delete failing browser tests Jenkins jobs - https://phabricator.wikimedia.org/T94150#1778044 (10SBisson) [17:41:47] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow, 5Patch-For-Review: Fix or delete failing Flow browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94153#1778042 (10SBisson) 5Open>3Resolved Flow browser tests are being monitored on a daily basis for regression and instability. Quick chan... [17:45:20] 10Continuous-Integration-Config, 10pywikibot-core: jenkins output is unreadable - https://phabricator.wikimedia.org/T117570#1778066 (10Legoktm) [17:45:35] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Echo, 10Flow: mediawiki_api client.create_page fails on Flow board - https://phabricator.wikimedia.org/T71321#1778069 (10SBisson) 5Open>3Invalid a:3SBisson Not relevant anymore with the current state of those tests. The echo tests are being completely r... [17:47:52] 10Browser-Tests, 10Flow, 3Collaboration-Team-Current, 7WorkType-NewFunctionality: Make Flow browser tests stable - https://phabricator.wikimedia.org/T109785#1778081 (10SBisson) 5Open>3declined a:3SBisson This is being worked on on a daily basis. [17:48:02] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow, 7WorkType-NewFunctionality: Make Flow browser tests stable - https://phabricator.wikimedia.org/T109785#1778084 (10SBisson) [17:57:42] krenair@mira:/srv/mediawiki-staging$ git diff [17:57:42] fatal: Not a git repository: /mnt/srv/mediawiki-staging/.git/modules/portals [17:58:28] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow, 7Easy: send Echo and Flow (and any Collaboration team extensions with browser tests) browser test job notices to #wikimedia-collaboration channel - https://phabricator.wikimedia.org/T66103#1778114 (10dduvall) Is this still desired by Collaboration? IRC... [18:05:45] 10Browser-Tests, 6Collaboration-Team-Backlog, 10Flow: Sauce Labs screencast for "No JavaScript" Flow tests shows empty browser window. - https://phabricator.wikimedia.org/T86707#1778133 (10dduvall) 5Open>3Resolved a:3dduvall The description and test history suggests an intermittent issue with SauceLabs... [18:07:27] I think scap is broken on beta [18:07:49] I synchronised a file on deployment-bastion and it didn't go to the local /srv/mediawiki [18:12:49] 10Beta-Cluster-Infrastructure: scap broken on beta? - https://phabricator.wikimedia.org/T117574#1778153 (10Krenair) 3NEW [18:17:32] (03CR) 10JanZerebecki: [C: 032] "Deployed to Jenkins: ['browsertests-CentralAuth-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce', 'browsertests-CentralNotice-en.wikiped" [integration/config] - 10https://gerrit.wikimedia.org/r/247901 (https://phabricator.wikimedia.org/T116166) (owner: 10JanZerebecki) [18:22:38] (03Merged) 10jenkins-bot: Additionally run Wikidata browsertests without saucelabs [integration/config] - 10https://gerrit.wikimedia.org/r/247901 (https://phabricator.wikimedia.org/T116166) (owner: 10JanZerebecki) [18:27:40] RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 2.09 ms [18:31:09] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: [Task] investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1778237 (10JanZerebecki) [18:31:12] 10Browser-Tests, 10Wikidata, 7Easy, 5Patch-For-Review, and 2 others: move wikidata browsertests to not use saucelabs - https://phabricator.wikimedia.org/T116166#1778234 (10JanZerebecki) 5Open>3Resolved a:3JanZerebecki Now the old jobs are switched back to saucelabs again and new jobs are created that... [18:32:37] 10Browser-Tests, 10Wikidata, 3Wikidata-Sprint-2015-11-03: selenium fails to connect to firefox (headless not sauce) - https://phabricator.wikimedia.org/T117561#1778245 (10JanZerebecki) [18:41:29] (03PS1) 10JanZerebecki: Correct Wikidata browsertest job names [integration/config] - 10https://gerrit.wikimedia.org/r/250718 [18:48:11] (03CR) 10JanZerebecki: [C: 032] "Deployed Jenkins jobs: ['browsertests-Wikidata-WikidataTests-linux-chrome-sauce', 'browsertests-Wikidata-WikidataTests-linux-firefox', 'br" [integration/config] - 10https://gerrit.wikimedia.org/r/250718 (owner: 10JanZerebecki) [18:59:30] (03Merged) 10jenkins-bot: Correct Wikidata browsertest job names [integration/config] - 10https://gerrit.wikimedia.org/r/250718 (owner: 10JanZerebecki) [19:04:05] Project browsertests-Wikidata-SmokeTests-linux-firefox build #1: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox/1/ [19:05:29] 10Browser-Tests, 10Wikidata, 7Regression, 3Wikidata-Sprint-2015-11-03: AuthorityControl gadget browsertest fail - https://phabricator.wikimedia.org/T117564#1778433 (10JanZerebecki) 5Open>3Resolved [19:05:49] 10Browser-Tests, 10Wikidata, 7Regression, 3Wikidata-Sprint-2015-11-03: AuthorityControl gadget browsertest fail - https://phabricator.wikimedia.org/T117564#1777775 (10JanZerebecki) Yes the browser test works now, too. [19:14:57] 10Browser-Tests, 10Wikidata: Wikidata Feature: Item smoke test: fails to find cancel button - https://phabricator.wikimedia.org/T117582#1778473 (10JanZerebecki) 3NEW [19:15:18] 10Browser-Tests, 10Wikidata: Wikidata Feature: Item smoke test: fails to find cancel button - https://phabricator.wikimedia.org/T117582#1778481 (10JanZerebecki) [19:15:19] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: [Task] investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1778480 (10JanZerebecki) [19:23:28] 10Browser-Tests, 10Wikidata: Wikidata Feature: Edit label: goes to the non-js version - https://phabricator.wikimedia.org/T117584#1778519 (10JanZerebecki) 3NEW [19:28:10] 10Browser-Tests, 10Wikidata: Wikidata various features: go to the non-js version of edit label, description, alias - https://phabricator.wikimedia.org/T117584#1778539 (10JanZerebecki) [19:36:55] 10Deployment-Systems, 6Performance-Team, 6operations, 7Epic, 7Tracking: During deployment old servers may populate new cache URIs (tracking) - https://phabricator.wikimedia.org/T47877#1778584 (10Krinkle) [19:42:42] 10Browser-Tests, 10Wikidata: Wikidata Smoketest Headless: XHR didn't load - https://phabricator.wikimedia.org/T117591#1778679 (10JanZerebecki) 3NEW [19:43:26] 10Browser-Tests, 10Wikidata: Wikidata various features: edit label, description, alias goes to the non-js version - https://phabricator.wikimedia.org/T117584#1778693 (10JanZerebecki) [19:48:15] 10Browser-Tests, 10Wikidata: Wikidata various features: edit label, description, alias goes to the non-js version - https://phabricator.wikimedia.org/T117584#1778730 (10JanZerebecki) [19:52:22] 10Browser-Tests, 10Wikidata: Wikidata various features: edit label, description, alias goes to the non-js version - https://phabricator.wikimedia.org/T117584#1778755 (10aude) this happens to me also on beta. this is because the js is quite slow in loading sometimes and if you are too quick to click [edit] the... [19:59:10] 10Continuous-Integration-Config, 10pywikibot-core: jenkins output is unreadable - https://phabricator.wikimedia.org/T117570#1778778 (10XZise) The duplicates are because we have flake8 for Python 2 (`flake8`) and Python 3 (`flake8-py3`) and a set of more strict rules (`flake8-docstrings-mandatory`). And for exa... [20:13:33] greg-g: i apparently need to deploy an extension this week https://phabricator.wikimedia.org/T110661 - would there be a good day to do this? [20:24:55] 10Continuous-Integration-Config, 10pywikibot-core: jenkins output is unreadable - https://phabricator.wikimedia.org/T117570#1778969 (10valhallasw) ``` valhallasw@maeglin:pywikibot-core$ grep consoleText -P -e ':\d+:\d+:\s[A-Z]+[0-9]+' | sed -e 's:^./::' | sort | uniq pywikibot/__init__.py:469:1: E303 too many... [20:50:32] 10Browser-Tests, 10Wikidata: Wikidata various features: edit label, description, alias goes to the non-js version - https://phabricator.wikimedia.org/T117584#1779055 (10Mbch331) Also happens in production for the same reason as aude mentions in his comment. [20:56:03] 10Beta-Cluster-Infrastructure: scap broken on beta? - https://phabricator.wikimedia.org/T117574#1779070 (10hashar) 5Open>3Resolved a:3hashar The Jenkins job is all happy https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ [20:57:18] 10Beta-Cluster-Infrastructure: scap broken on beta? - https://phabricator.wikimedia.org/T117574#1779074 (10Krenair) 5Resolved>3Open Yes, and the command I ran was happy. Doesn't mean it actually worked. [20:59:19] 10Beta-Cluster-Infrastructure: scap broken on beta? - https://phabricator.wikimedia.org/T117574#1779079 (10Krenair) ```jenkins-deploy@deployment-bastion:/mnt/srv/mediawiki-staging$ git diff diff --git a/wmf-config/CommonSettings.php b/wmf-config/CommonSettings.php index d59c6d2..87ab7fd 100755 --- a/wmf-config/C... [20:59:57] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: scap broken on beta? - https://phabricator.wikimedia.org/T117574#1779085 (10Krenair) [21:07:02] PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145) [21:10:25] thcipriani: twentyafterfour ostriches any idea about that scap fail task ^ [21:10:38] I was looking, nothing jumped out immediately. [21:10:57] * ostriches grumbles about just now getting back home, stupid muni [21:11:00] looking into it now, oddly seems limited to mira and the deployment host. Other hosts seem to be getting the commonsettings.php update. [21:11:25] I wonder if it relates to the permission problem since it's only mira/bastion. [21:11:38] (for which a patch just landed, I think) [21:11:46] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #817: 04FAILURE in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/817/ [21:13:19] hmmm [21:13:25] I _think_ that change only affected the /srv/mediawiki-staging directory so it wouldn't totally explain the failure on deployment-bastion. [21:13:58] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779160 (10hashar) a:5hashar>3None [21:14:16] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1778153 (10hashar) [21:15:11] and, actually, mira has the update in /srv/mediawiki-staging, just not in /srv/mediawiki [21:17:43] That's doubly weird. [21:20:02] what's the permissions update? [21:20:42] https://gerrit.wikimedia.org/r/#/c/249684/ [21:20:54] Erm, there was another one [21:20:55] Hmm [21:21:55] Ah no that was it [21:22:00] Commit msg confused me somehow [21:23:16] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779209 (10hashar) That is a better report, I have updated the task details. Might be caused by https://gerrit.wikimedia.org/r/#/c/224313/ `Sync /srv/m... [21:23:33] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779214 (10hashar) p:5Triage>3High [21:27:54] 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-November-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1779345 (10Qgil) a:3Qgil I guess I'll take this task. @greg, what about making it a blocker of the release task? [21:28:48] hmm, so it's somehow not getting to the rsync command, because the actual rsync command works on deployment-bastion [21:29:10] sudo -u mwdeploy -n -- /usr/bin/rsync --archive --delete-delay --delay-updates --compress --delete --exclude=**/cache/l10n/*.cdb --no-perms --exclude=**/.git --exclude=* --include=/wmf-config/CommonSettings.php deployment-bastion.eqiad.wmflabs::common /srv/mediawik [21:29:22] i [21:29:55] which is strange since I tried that with sync-file first and it didn't seem to sync /srv/mediawiki [21:37:27] Never seen it fail like this hmm [21:39:34] chasemp: I can do that repo swap once we're done figuring this out [21:39:48] Don't wanna move more parts while we're busted [21:39:51] ostriches: just ping me when ready :) [21:41:07] so, I'm not sure this is a "new" thing: https://gerrit.wikimedia.org/r/#/c/135588/1/scap/main.py [21:41:19] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779373 (10bd808) The local /srv/mediawiki directory on the deployment server is [[https://phabricator.wikimedia.org/diffusion/MSCA/browse/master/scap/ma... [21:41:46] ^ bing! that was what that commit suggested :) [21:43:16] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779377 (10bd808) >>! In T117574#1779209, @hashar wrote: > Line 62 has `update_masters.exclude_hosts([socket.getfqdn()])` which looks like sync-masters d... [21:43:35] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779384 (10Krenair) Not in production: ```krenair@tin:/srv/mediawiki-staging (master)$ nano README krenair@tin:/srv/mediawiki-staging (master)$ sync-file... [21:44:36] I love how everyone talking about this is in the same channel but we're doing it async via the task, heh. [21:45:02] the "bug" is the dsh group [21:48:08] 10Beta-Cluster-Infrastructure, 10Deployment-Systems: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779420 (10bd808) Tin is in >>! In T117574#1779384, @Krenair wrote: > Not in production: `tin.eqiad.wmnet` is in /etc/dsh/group/mediawiki-installation... [21:51:36] * bd808 will cherry-pick [21:52:01] Krenair: yeah it probably needs mira too [21:54:08] I wonder if we actually need it in the list [21:54:27] Or if we could just join it with the scap_masters and then unique dupes out. [21:54:29] if it's not there then /srv/mediawiki won't be updated [21:54:38] yeah that could be done [21:54:52] and probably will need to be when you switch to getting the hosts from etcd [21:55:07] but gawd knows how that will be handled for beta cluster [21:55:14] (no pybal) [21:55:14] I'll go ahead and work up a patch for that at least. [21:55:17] Should be easy. [21:58:46] why is puppet so sloooow on deployment-bastion? [21:58:58] !log cherry-picked https://gerrit.wikimedia.org/r/#/c/250837/ and forced puppet run on deployment-bastion [21:59:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:01:26] ssh keys aren't setup right too :( [22:01:45] 22:00:57 ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n', '--include', 'README'] on deployment-bastion.deployment-prep.eqiad.wmflabs returned [255]: Warning: Permanently added 'deployment-bastion.deployment-prep.eqiad.wmflabs,10.68.16.58' (ECDSA) to the list of known hosts. [22:01:46] Connection closed by 10.68.16.58 [22:03:55] * bd808 pokes about in wikitech [22:05:22] might guess it still has deployment-bastion.eqiad.wmflabs rather than deployment-bastion.deployment-prep.eqiad.wmflabs [22:05:47] !log applied ::beta::deployaccess on deployment-bastion via Special:NovaInstance [22:05:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:06:29] thcipriani: just needed the right role added [22:06:54] gotcha. [22:08:23] * bd808 twiddles while puppet runs [22:09:47] if you are only trying to update one thing you can use a tag too from cli [22:09:58] compiles a full catalogue but may be quicker :) [22:11:21] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779614 (10bd808) ``` jenkins-deploy@deployment-bastion:/srv/mediawiki-staging$ vim README $ git diff README diff --git a/README b/R... [22:12:42] Project beta-scap-eqiad build #77150: 04FAILURE in 8 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/77150/ [22:13:01] chasemp: want to merge https://gerrit.wikimedia.org/r/#/c/250837/ for us? [22:13:34] sure [22:14:42] slow jenkins [22:15:58] (03PS6) 10Paladox: [EventLogging] Add jsduck test [integration/config] - 10https://gerrit.wikimedia.org/r/246773 (https://phabricator.wikimedia.org/T88343) [22:16:07] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779634 (10bd808) 5Open>3Resolved a:3bd808 [22:16:12] thanks chasemp [22:17:03] (03CR) 10Paladox: "Patch was merged. So this can now be merged once https://gerrit.wikimedia.org/r/#/c/246841/ is merged." [integration/config] - 10https://gerrit.wikimedia.org/r/246773 (https://phabricator.wikimedia.org/T88343) (owner: 10Paladox) [22:17:10] (03PS7) 10Paladox: [EventLogging] Add jsduck test [integration/config] - 10https://gerrit.wikimedia.org/r/246773 (https://phabricator.wikimedia.org/T88343) [22:17:13] good catch Krenair. looks like that has been busted since forever [22:17:25] :) [22:17:26] (03PS8) 10Paladox: [EventLogging] Add jsduck test [integration/config] - 10https://gerrit.wikimedia.org/r/246773 (https://phabricator.wikimedia.org/T88343) [22:18:34] 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review: scap on beta does not sync deployment-bastion /srv/mediawiki - https://phabricator.wikimedia.org/T117574#1779640 (10demon) [22:19:28] D33 also up so we can avoid this again. [22:20:26] Yippee, build fixed! [22:20:26] Project beta-scap-eqiad build #77151: 09FIXED in 5 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/77151/ [22:20:39] (03PS3) 10Paladox: [WebPlatformAuth] Update Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/246712 (https://phabricator.wikimedia.org/T115061) [22:21:48] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/243394 (owner: 10Paladox) [22:22:05] (03CR) 10jenkins-bot: [V: 04-1] [cldr] Update jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/243394 (owner: 10Paladox) [22:22:15] RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms [22:22:57] Can't we just kill parsoidcache02 now? [22:23:03] I thought we already pointed at 05. [22:24:00] chasemp: Ok, I think we're clear and can do https://gerrit.wikimedia.org/r/#/c/250578/ now [22:24:16] (03PS2) 10Paladox: [cldr] Update jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/243394 [22:24:23] Merge, puppet run on tin/mira, then I'll fix up the git remotes. [22:26:16] marxarelli: around? [22:26:32] I could do with some help working out what to do with https://gerrit.wikimedia.org/r/#/c/246801/ [22:26:36] jdlrobson: meeting atm, but i'll be free soonish [22:29:29] ostriches: ok off I go [22:31:21] go go gadget! [22:32:01] gtg [22:32:20] I'm not going to run puppet wherever manually tho [22:32:39] ...wherever that may be applicable if it is :) [22:33:10] It should be fine, really. The remotes are mainly what need updating. [22:36:20] Actually, it's just the remote on tin. Mira points to tin, as it should. [22:37:00] !log deployment-bastion: scap now pointing to Phab repo instead of Gerrit. [22:37:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:37:15] ostriches: we did it diffusion is real [22:37:32] ostriches: yay [22:37:52] Also marked the repo in Gerrit r/o. [22:38:38] And the Phab repo is now hosted instead of mirroring. [22:39:12] kudos to you guys for shaking out the pilot on this [22:39:22] it's the organic thing that has to happen and can't be forced [22:42:29] (03PS1) 10Thcipriani: mw-tools-scap doc generation listen for publish [integration/config] - 10https://gerrit.wikimedia.org/r/250847 [22:44:48] (03PS2) 10Dduvall: Generate mw-tools-scap docs on publish [integration/config] - 10https://gerrit.wikimedia.org/r/250847 (owner: 10Thcipriani) [22:44:58] (03CR) 10Dduvall: [C: 032] Generate mw-tools-scap docs on publish [integration/config] - 10https://gerrit.wikimedia.org/r/250847 (owner: 10Thcipriani) [22:45:32] 10MediaWiki-Releasing, 6Developer-Relations, 10Wikimedia-Blog-Content, 3DevRel-November-2015, 5MW-1.26-release: Write blog post announcing MW 1.26 - https://phabricator.wikimedia.org/T112842#1779715 (10greg) It's in #MW-1.26-release, which means it's a blocker :) [22:46:23] (03Merged) 10jenkins-bot: Generate mw-tools-scap docs on publish [integration/config] - 10https://gerrit.wikimedia.org/r/250847 (owner: 10Thcipriani) [22:48:41] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/250847 [22:48:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:50:42] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/240326 (owner: 10Paladox) [22:51:38] (03CR) 10jenkins-bot: [V: 04-1] [Translate] Add composer-test test [integration/config] - 10https://gerrit.wikimedia.org/r/240326 (owner: 10Paladox) [22:53:41] When does publish run? [22:54:08] thcipriani: ^ [22:54:31] ostriches: was just looking at that: https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation [22:55:01] (03PS2) 10Paladox: [Translate] Add composer-test test [integration/config] - 10https://gerrit.wikimedia.org/r/240326 [22:55:28] thcipriani: Considering we're now read-only in gerrit, we should probably just scrap it from zuul. [22:55:39] And just put in in jjb as a timed job. [23:01:02] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #850: 04FAILURE in 50 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/850/ [23:01:32] Ugh, our long tail in #logspam is at 97 :( [23:01:37] Er, 96. [23:01:40] Still [23:02:11] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/239681 (owner: 10Paladox) [23:10:19] ostriches: publish runs whenever a tag is pushed [23:10:37] Yeah, but we don't push to gerrit no more no more. [23:10:52] trigger: [23:10:52] gerrit: [23:10:52] - event: ref-updated [23:10:53] ref: ^refs/tags/.*$ [23:11:34] I thought the grand plan was to set up a phabricator trigger? [23:13:07] oh, I guess we're switching to harbormaster? [23:13:22] https://phabricator.wikimedia.org/T103127 [23:17:10] https://www.mediawiki.org/wiki/MediaWiki_1.27/Roadmap is missing near future again :( [23:25:15] It looks like updating that page has been left to Florian [23:30:19] or, maybe, the AGF interpretation is: Since florian was doing a good job updating it others stopped [23:30:31] but, whatever, we can bitch too [23:31:24] MatmaRex: what info are you needing? [23:49:27] (03PS2) 10Paladox: [LinkSuggest2] Update jenkings test to more advanced tests [integration/config] - 10https://gerrit.wikimedia.org/r/239681 [23:49:36] (03PS3) 10Paladox: [LinkSuggest2] Update Jenkins test to more advanced tests [integration/config] - 10https://gerrit.wikimedia.org/r/239681 [23:53:49] PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)