[02:01:53] Yippee, build fixed! [02:01:54] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #318: FIXED in 36 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/318/ [03:35:06] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #108: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/108/ [04:55:29] Project beta-scap-eqiad build #32018: FAILURE in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32018/ [05:15:24] Yippee, build fixed! [05:15:25] Project beta-scap-eqiad build #32020: FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32020/ [05:23:15] Yippee, build fixed! [05:23:15] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #147: FIXED in 55 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/147/ [06:06:33] Yippee, build fixed! [06:06:33] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #356: FIXED in 55 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/356/ [06:34:26] Yippee, build fixed! [06:34:26] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #441: FIXED in 1 hr 11 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/441/ [07:16:28] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #362: FAILURE in 1 hr 9 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/362/ [08:55:20] (03PS2) 10Hashar: Add flake8 for server/bin of mediawiki/extensions/EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/176399 (owner: 10QChris) [08:56:54] (03CR) 10Hashar: [C: 032] "I looked at the EventLogging repo, that additionally flake8 env make sense there :-] Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/176399 (owner: 10QChris) [09:00:24] (03Merged) 10jenkins-bot: Add flake8 for server/bin of mediawiki/extensions/EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/176399 (owner: 10QChris) [09:09:30] (03PS4) 10Hashar: Move wikidata jobs to regular labs slaves. [integration/config] - 10https://gerrit.wikimedia.org/r/176270 (owner: 10JanZerebecki) [09:10:43] (03CR) 10Hashar: [C: 032] "Danke schön Jan! Seems all jobs are happy and we can now delete the old instances." [integration/config] - 10https://gerrit.wikimedia.org/r/176270 (owner: 10JanZerebecki) [09:14:18] (03Merged) 10jenkins-bot: Move wikidata jobs to regular labs slaves. [integration/config] - 10https://gerrit.wikimedia.org/r/176270 (owner: 10JanZerebecki) [09:28:03] 3Continuous-Integration: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#795992 (10hashar) I have emailed the ops list to attract some reviewers: ---- Hello, The wiki Apache conf has been moved to puppet during the summer and since then the Jenk... [09:30:04] 3Continuous-Integration: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#795993 (10hashar) a:5hashar>3None [09:30:48] 3Beta-Cluster, Wikimedia-Git-or-Gerrit, MediaWiki-General-or-Unknown: Create ability to trivially spin up MediaWiki instance of a particular Gerrit changeset - https://phabricator.wikimedia.org/T76245#795995 (10hashar) [09:32:21] 3Continuous-Integration: make jobs that depend on wikidata jenkins slaves run on the normal labs slaves instead - https://phabricator.wikimedia.org/T73419#795997 (10hashar) a:3JanZerebecki [09:33:40] 3Continuous-Integration: make jobs that depend on wikidata jenkins slaves run on the normal labs slaves instead - https://phabricator.wikimedia.org/T73419#796000 (10hashar) 5Open>3Resolved Solved by @JanZerebecky by https://gerrit.wikimedia.org/r/176270 . Danke schön Jan! [10:11:07] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #36: FAILURE in 2 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/36/ [10:50:45] 3Continuous-Integration: Delete legacy wikidata-jenkins* slaves - https://phabricator.wikimedia.org/T76333 (10hashar) 3NEW p:3Triage [11:15:27] Project beta-scap-eqiad build #32055: FAILURE in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32055/ [11:26:56] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #337: FAILURE in 1 h 19 mn: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/337/ [11:40:00] (03PS1) 10Hashar: Limit browser tests history to 31 days [integration/config] - 10https://gerrit.wikimedia.org/r/176649 [11:40:18] (03PS2) 10Hashar: Limit browser tests history to 31 days [integration/config] - 10https://gerrit.wikimedia.org/r/176649 [11:40:38] (03PS3) 10Hashar: Limit browser tests history to 31 days [integration/config] - 10https://gerrit.wikimedia.org/r/176649 [11:45:11] (03CR) 10Hashar: "We probably want to have this approved by browser tests shareholders. Not sure whom to contact beside the persons I have added already, I " [integration/config] - 10https://gerrit.wikimedia.org/r/176649 (owner: 10Hashar) [11:47:13] (03CR) 10KartikMistry: Limit browser tests history to 31 days (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/176649 (owner: 10Hashar) [11:50:16] hashar: Any suggestion for https://gerrit.wikimedia.org/r/175979 ? [11:54:47] 3Release-Engineering: Add in Phabricator quarterly milestones for RelEng - https://phabricator.wikimedia.org/T75729#796176 (10Qgil) p:5Triage>3Normal [11:57:58] 3Release-Engineering, Continuous-Integration: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#796185 (10hashar) [11:58:11] 3Release-Engineering, Continuous-Integration: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#747903 (10hashar) [11:59:20] 3Wikimedia-Wikidata, Continuous-Integration: jenkins job time outs - https://phabricator.wikimedia.org/T76130#796187 (10hashar) The Wikidata jobs have been migrated to the Wikimedia Jenkins slaves (integration-slave*). [12:01:34] Yippee, build fixed! [12:01:35] Project beta-scap-eqiad build #32057: FIXED in 27 mn: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32057/ [12:16:15] (03CR) 10Hashar: Limit browser tests history to 31 days (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/176649 (owner: 10Hashar) [13:07:37] (03CR) 10KartikMistry: [C: 031] Limit browser tests history to 31 days [integration/config] - 10https://gerrit.wikimedia.org/r/176649 (owner: 10Hashar) [13:08:00] (03CR) 10Zfilipin: [C: 031] Limit browser tests history to 31 days [integration/config] - 10https://gerrit.wikimedia.org/r/176649 (owner: 10Hashar) [13:09:16] aharoni: around? [14:15:03] 3Release-Engineering, Continuous-Integration: Zuul-cloner forgets to clear workspace - https://phabricator.wikimedia.org/T76304#796299 (10hashar) [14:23:46] 3Wikimedia-Wikidata, Wikidata, Continuous-Integration: jenkins job time outs - https://phabricator.wikimedia.org/T76130#796357 (10Lydia_Pintscher) [14:59:43] 3Release-Engineering: Submit "A Self-Translating User Manual" talk to a couple of FOSDEM devrooms - https://phabricator.wikimedia.org/T1389#24335 (10zeljkofilipin) [15:38:29] 3Continuous-Integration: zuul-cloner does not support Ref events - https://phabricator.wikimedia.org/T76003#798457 (10hashar) [16:02:39] Yippee, build fixed! [16:02:39] Project beta-mediawiki-config-update-eqiad build #1510: FIXED in 1,5 s: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/1510/ [16:09:28] Yippee, build fixed! [16:09:28] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #37: FIXED in 2 min 12 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/37/ [16:37:30] hi hashar is "npm dependencies" something that you know about? https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-npm/168/consoleFull [16:42:47] chrismcmahon: hashar: jenkins npm task is failing on npm install in several patches https://gerrit.wikimedia.org/r/173025 and https://gerrit.wikimedia.org/r/#/c/173025/ , for example, it makes no sense. Here is the output https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-npm/167/console , any ideas? [16:57:52] joakino: fill a task in Phabricator please with project #contint [16:58:03] please :-] [16:58:22] hashar: sure, is it just us? :S [16:58:51] might be https://phabricator.wikimedia.org/T76304 [16:59:00] the npm jobs workspaces are not cleared between runs [16:59:08] we might need to add "npm prune" [17:01:04] hashar: oki, i'll open a task and reference that issue [17:01:16] and poke Krinkle|detached about it ^^^ :-] [17:02:15] 👍 [17:03:10] 3Wikidata, Continuous-Integration: Delete legacy wikidata-jenkins* slaves - https://phabricator.wikimedia.org/T76333#798566 (10JanZerebecki) I want to wait a bit to make sure they are not needed anymore. Then, after you removed them from jenkins, i'll delete the instances. @hoo @aude Do we still need the instan... [17:03:34] 3Wikidata, Continuous-Integration: Delete legacy wikidata-jenkins* slaves - https://phabricator.wikimedia.org/T76333#798569 (10JanZerebecki) [17:06:50] 3Wikidata, Continuous-Integration: Delete legacy wikidata-jenkins* slaves - https://phabricator.wikimedia.org/T76333#798579 (10hashar) [17:07:35] 3Continuous-Integration: [OPS] Jenkins: Slaves running Ubuntu Trusty should have hhvm installed - https://phabricator.wikimedia.org/T75356#798580 (10hashar) I have poked the ops list to get help on installing hhvm and properly configure it on Jenkins slaves. [17:08:09] 3MobileFrontend-General-or-Unknown, Continuous-Integration: mwext-MobileFrontend-npm is failing to complete on several commits (not verifying commits, can't merge) - https://phabricator.wikimedia.org/T76354 (10Jhernandez) 3NEW p:3Triage [17:08:24] 3Continuous-Integration: Jenkins: Implement mediawiki-phpunit-hhvm job for mediawiki-core repo (and voting) - https://phabricator.wikimedia.org/T75521#798592 (10hashar) [17:15:12] hashar: Krinkle|detached: when you can, have a look at https://phabricator.wikimedia.org/T76354 , please. thank you [17:21:23] 3MobileFrontend-General-or-Unknown, Continuous-Integration: mwext-MobileFrontend-npm is failing to complete on several commits (not verifying commits, can't merge) - https://phabricator.wikimedia.org/T76354#798612 (10Jhernandez) [17:22:14] Yippee, build fixed! [17:22:14] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #363: FIXED in 1 hr 14 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/363/ [17:38:39] 3MobileFrontend-General-or-Unknown, Continuous-Integration: mwext-MobileFrontend-npm is failing to complete on several commits (not verifying commits, can't merge) - https://phabricator.wikimedia.org/T76354#798644 (10Jhernandez) It seems like npm tasks executed on integration-slave1006 fail for various reasons.... [17:45:20] 3MobileFrontend-General-or-Unknown, Continuous-Integration: mwext-MobileFrontend-npm is failing to complete on several commits (not verifying commits, can't merge) - https://phabricator.wikimedia.org/T76354#798650 (10Florian) p:5Triage>3High It seems, that tests, triggered by integration-slave1006 fails. See... [17:53:13] (03CR) 10Jforrester: [C: 031] Limit browser tests history to 31 days [integration/config] - 10https://gerrit.wikimedia.org/r/176649 (owner: 10Hashar) [17:59:28] !log Testing rsyslog event forwarding to logstash via puppet cherry-pick [17:59:31] Logged the message, Master [18:09:22] joakino: checking out https://phabricator.wikimedia.org/T76354 after the VE standup. [18:15:48] Krinkle: 👍 thanks [18:24:39] hi marxarelli do we have a preferred RSpec syntax for string equality? e.g. is this OK? https://gerrit.wikimedia.org/r/#/c/176680/1/tests/browser/features/step_definitions/diff_steps.rb [18:28:35] 3Continuous-Integration, MobileFrontend-General-or-Unknown: mwext-MobileFrontend-npm is failing to complete on several commits (not verifying commits, can't merge) - https://phabricator.wikimedia.org/T76354#798742 (10Jhernandez) [18:42:42] 3Continuous-Integration, MobileFrontend-General-or-Unknown: mwext-MobileFrontend-npm is failing to complete on several commits (not verifying commits, can't merge) - https://phabricator.wikimedia.org/T76354#798794 (10Krinkle) [18:46:22] 3Continuous-Integration, MobileFrontend-General-or-Unknown: mwext-MobileFrontend-npm is failing to complete on several commits (not verifying commits, can't merge) - https://phabricator.wikimedia.org/T76354#798828 (10KLans_WMF) [18:51:01] !log updates in beta suck with the "Waiting for next available executor" deadlock again [18:51:03] Logged the message, Master [18:51:12] heh. stuck, but suck works too [18:54:25] !log Got jenkins updates working again by taking deployment-bastion node offline, killing waiting jobs and bringing it back online again. [18:54:27] Logged the message, Master [19:01:45] Project beta-scap-eqiad build #32084: FAILURE in 9.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32084/ [19:12:12] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #302: FAILURE in 32 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/302/ [19:12:30] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #148: FAILURE in 3.6 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/148/ [19:12:44] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #163: FAILURE in 8.1 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/163/ [19:12:48] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #218: FAILURE in 3.9 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/218/ [19:13:11] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #357: FAILURE in 22 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/357/ [19:13:20] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #442: FAILURE in 8.1 sec: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/442/ [19:13:26] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #256: FAILURE in 5.6 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/256/ [19:13:30] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #38: FAILURE in 3.9 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/38/ [19:13:35] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #335: FAILURE in 4.5 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/335/ [19:13:42] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #364: FAILURE in 6.5 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/364/ [19:13:46] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #292: FAILURE in 3.3 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/292/ [19:13:49] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #200: FAILURE in 3.2 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/200/ [19:14:07] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #294: FAILURE in 17 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/294/ [19:14:21] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #260: FAILURE in 14 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/260/ [19:14:42] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #398: FAILURE in 14 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/398/ [19:37:42] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:38:42] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:09:38] Yippee, build fixed! [20:09:39] Project beta-scap-eqiad build #32089: FIXED in 25 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32089/ [20:11:40] Project beta-scap-eqiad build #32090: FAILURE in 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32090/ [20:18:47] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #308: FAILURE in 16 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/308/ [20:19:09] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #318: FAILURE in 6.2 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/318/ [20:19:16] Project browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce build #198: FAILURE in 5.9 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-firefox-sauce/198/ [20:19:21] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #320: FAILURE in 4.8 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/320/ [20:19:47] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #38: FAILURE in 2.5 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/38/ [20:19:50] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #164: FAILURE in 15 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/164/ [20:19:57] Project browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #192: FAILURE in 8.5 sec: https://integration.wikimedia.org/ci/job/browsertests-GettingStarted-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/192/ [20:20:08] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #319: FAILURE in 11 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/319/ [20:20:15] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #320: FAILURE in 5.8 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/320/ [20:20:18] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #264: FAILURE in 3.4 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/264/ [20:20:19] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #314: FAILURE in 9.9 sec: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/314/ [20:55:41] marxarelli: are you around? I have an RSpec question for you... [20:55:50] chrismcmahon: sure [20:56:39] marxarelli: do we prefer "eq" for string equality checking, or some other syntax? https://gerrit.wikimedia.org/r/#/c/176680/1/tests/browser/features/step_definitions/diff_steps.rb [20:57:05] I read the RSpec docs and TIMTOWTDI [20:58:09] chrismcmahon: either `eq` or `==` is probably fine [20:59:19] chrismcmahon: i think `be` technically works for string comparison but should probably be avoided due to it using object identity elsewhere [21:02:00] marxarelli: sounds good, thanks, I'll merge that then after I fix the rebase [21:02:21] chrismcmahon: no prob [21:31:07] marxarelli: also just FYI, in that same test I did not know until now that watir-webdriver allows you to define your own page elements, this is pretty nifty: https://gerrit.wikimedia.org/r/#/c/176794/1/tests/browser/features/support/pages/diff_page.rb [21:32:03] marxarelli: the MobileFrontend UI has an elements "ins" and "del" within a div that turn green and red for insert/delete diffs [21:33:02] hey qa, the rubocop task in VE is broken. i think it's yours? https://gerrit.wikimedia.org/r/#/c/176396/2 [21:33:23] (broken = failing and falsely blocking all merges) [21:36:54] MatmaRex: when did that become voting? the problem seems to be that rubygems.org is down for some reason. [21:37:13] chrismcmahon: no idea. [21:40:25] 3Wikimedia-Git-or-Gerrit, Labs-Vagrant, MediaWiki-General-or-Unknown, Beta-Cluster: Create ability to trivially spin up MediaWiki instance of a particular Gerrit changeset - https://phabricator.wikimedia.org/T76245#799283 (10greg) I know some places that have a set number of "qa instances" that people can "check... [21:40:45] 3Wikimedia-Git-or-Gerrit, Labs-Vagrant, MediaWiki-General-or-Unknown, Release-Engineering: Create ability to trivially spin up MediaWiki instance of a particular Gerrit changeset - https://phabricator.wikimedia.org/T76245#799286 (10greg) [21:42:00] 3VisualEditor, Beta-Cluster: VisualEditor not updating in beta labs - https://phabricator.wikimedia.org/T76097#799289 (10greg) [21:43:03] 3Scrum-of-Scrums, Beta-Cluster: Create labs ldap user and group "apache" to be used for NFS4 access in deployment-prep project - https://phabricator.wikimedia.org/T76086#799294 (10greg) [21:44:33] marxarelli: also FYI, rubygems.org is down because of a massive DDoS on DNSimple right now it seems. this causes rubocop failures. [21:46:08] 3Wikimedia-Git-or-Gerrit, MediaWiki-Vagrant, Labs-Vagrant, MediaWiki-General-or-Unknown, Release-Engineering: Create ability to trivially spin up MediaWiki instance of a particular Gerrit changeset - https://phabricator.wikimedia.org/T76245#799303 (10bd808) This kind of testing is exactly what I expect developer... [21:46:16] 3Beta-Cluster: Thumbnail 404s get cached - https://phabricator.wikimedia.org/T69056#799306 (10greg) @tgr: I tried to reproduce, but it seems to work now. Can you confirm? Urls I tried: http://upload.wikimedia.org/wikipedia/commons/thumb/5/52/Giant_planes_comparison.svg/11px-Giant_planes_comparison.svg.png and h... [21:47:39] 3Beta-Cluster: Low-quality images are not rendered in beta - https://phabricator.wikimedia.org/T71757#799315 (10greg) >>! In T71757#746442, @hashar wrote: > I think that is handled by deployment-upload.eqiad.wmflabs which is more or less a simulation of the old media servers using nginx. IIRC thumb.php got copi... [21:49:22] 3Beta-Cluster: Properly rotate udp2log logs (restart daemon) - https://phabricator.wikimedia.org/T72112#799320 (10greg) [21:49:35] 3Beta-Cluster: Properly rotate udp2log logs (restart daemon) - https://phabricator.wikimedia.org/T72112#725555 (10greg) p:5Triage>3Normal [21:49:50] 3Beta-Cluster: Properly rotate udp2log logs (restart daemon) - https://phabricator.wikimedia.org/T72112#725555 (10greg) [21:50:56] 3MediaWiki-extensions-CentralAuth, Beta-Cluster: CentralAuth logging a lot (login attempts/successes/etc) on Beta Cluster - https://phabricator.wikimedia.org/T75929#799329 (10greg) 5Open>3declined a:3greg >>! In T75929#786170, @Legoktm wrote: > All of those are logged in prod as well. The migration stuff h... [21:52:36] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#799334 (10greg) >>! In T75947#786812, @bd808 wrote: > I think this is fixed for now, but I'm still not 100% sure how things got messed up in the first place. Maybe @yuvipanda has some ideas? @Yuvipanda thoughts? [21:52:59] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#799339 (10greg) p:5Triage>3Low [21:54:28] 3Wikimedia-Git-or-Gerrit, MediaWiki-Vagrant, Labs-Vagrant, MediaWiki-General-or-Unknown, Release-Engineering: Create ability to trivially spin up MediaWiki instance of a particular Gerrit changeset - https://phabricator.wikimedia.org/T76245#799343 (10greg) [21:54:39] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#799344 (10yuvipanda) Nope, I found things in the same state earlier, and I re-applied the local hacks myself. My theory is that someone is testing things on betacluster who does not know about the local cherry... [21:56:33] 3Wikimedia-Git-or-Gerrit, MediaWiki-Vagrant, Labs-Vagrant, MediaWiki-General-or-Unknown, Release-Engineering: Create ability to trivially spin up MediaWiki instance of a particular Gerrit changeset - https://phabricator.wikimedia.org/T76245#799352 (10yuvipanda) So, I've thought about this a lot - essentially a '... [21:58:10] chrismcmahon: rubocop is still non-voting, right? [21:58:42] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#799353 (10greg) [21:59:02] marxarelli: apparently it is voting for this build https://gerrit.wikimedia.org/r/#/c/176396/2 (but I have no idea why that build even has rubocop on it) [22:00:07] 3Beta-Cluster: Local cherry-picks missing on deployment-salt - https://phabricator.wikimedia.org/T75947#786742 (10greg) >>! In T75947#799344, @yuvipanda wrote: > Nope, I found things in the same state earlier, and I re-applied the local hacks myself. My theory is that someone is testing things on betacluster who... [22:00:08] 3Beta-Cluster: shell wrapper to connect to databases - https://phabricator.wikimedia.org/T47706#799368 (10yuvipanda) Hmm, I wonder if the solution is not to just create a wikiadmin user on betacluster, and have things use that rather than the 'mw' user? [22:00:46] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#799372 (10yuvipanda) [22:01:07] 3Beta-Cluster: shell wrapper to connect to databases - https://phabricator.wikimedia.org/T47706#799374 (10yuvipanda) 5Resolved>3Open [22:01:21] 3Beta-Cluster: shell wrapper to connect to databases - https://phabricator.wikimedia.org/T47706#488886 (10yuvipanda) Re-opening, we should fix it by fixing the users, not the script. [22:02:27] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#799380 (10hashar) [22:02:40] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#799381 (10Krenair) [22:03:49] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#799383 (10hashar) [22:06:35] 3Beta-Cluster: shell wrapper to connect to databases - https://phabricator.wikimedia.org/T47706#799390 (10hashar) A workaround is currently applied for this on the beta cluster has a local hack: [LOCAL HACK] T47706 Change MySQL admin user in sql script ``` files/misc/scripts/sql -MU=wikiadmin +MU=mw ``` [22:08:02] 3Beta-Cluster: shell wrapper to connect to databases - https://phabricator.wikimedia.org/T47706#799395 (10greg) [22:08:05] 3Beta-Cluster: mwdeploy user has shell /bin/bash in labs LDAP and /bin/false in production/Puppet - https://phabricator.wikimedia.org/T67591#799397 (10hashar) [22:08:08] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#799398 (10greg) [22:08:53] 3Release-Engineering, Beta-Cluster: Reduce [LOCAL HACK] changes on Beta Cluster to zero - https://phabricator.wikimedia.org/T76392#799403 (10hashar) [22:12:15] 3Wikimedia-Logstash, Release-Engineering: Upgrade logstash to 1.4.2 - https://phabricator.wikimedia.org/T4#799424 (10Aklapper) [22:15:00] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:18:32] chrismcmahon: i suspect any job that executes through bundler is going to fail. one option might be to modify the `bundle` job to try `bundle install --local` first [22:19:08] that will avoid any connection to rubygems, using locally cached gems if possible [22:20:02] * marxarelli wishes he knew more about jenkins and how to test such hypotheses [22:23:12] marxarelli: feel free to dig in to it ;) [22:25:16] greg-g: on it :) i know enough to push a integration/config commit at least [22:25:19] marxarelli: we should really have a local gem repo if we don't already [22:25:36] * chrismcmahon thought we'd already done that at least once [22:40:03] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:59] (03PS1) 10Dduvall: Try local gem cache in bundle job [integration/config] - 10https://gerrit.wikimedia.org/r/176818 [22:50:53] greg-g: ^ who else should i add for review? [22:50:55] (03CR) 10Cmcmahon: [C: 031] "Let's not go to the internet if we don't need to." [integration/config] - 10https://gerrit.wikimedia.org/r/176818 (owner: 10Dduvall) [22:52:10] marxarelli: hmmmm [22:52:24] timo? [22:59:51] [23:02:52] marxarelli: yes? [23:03:25] Krinkle: hey! just looking for review of https://gerrit.wikimedia.org/r/#/c/176818/ [23:04:02] Krinkle: jobs that execute through bundler are failing due to a ddos attack on rubygems.org [23:04:20] marxarelli: Yeah [23:04:24] Krinkle: hoping to mitigate failures with that patch [23:05:27] 3Beta-Cluster: shell wrapper to connect to databases - https://phabricator.wikimedia.org/T47706#799596 (10hashar) >>! In T47706#799368, @yuvipanda wrote: > Hmm, I wonder if the solution is not to just create a wikiadmin user on betacluster, and have things use that rather than the 'mw' user? That is probably th... [23:05:30] marxarelli: putting it in a sibling dir of a workspace won't make it much better though. That's cleared from time to time. It should probably be put in the home directory of the jenkin-slave user (/mnt/home/jenkins-slave) similar to what we already do for npm [23:06:01] And it kind of violates the basic definitino of the directory: /mnt/jenkins-workspace/"workspaces" [23:06:17] but we can move that later [23:07:22] Krinkle: ah, ok. i haven't introduced use of that subdirectory, but i can certainly change it to another location that might be more appropriate [23:07:28] marxarelli: Does GEM_PATH need to be set for the gem2.0 commands? [23:07:38] Krinkle: i know very little about the Jekins/Zuul env [23:08:03] Krinkle: i believe it does, at least for `gem which` [23:08:48] marxarelli: OK. I'll compile locally and push one job to test [23:09:12] I'm following https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#Deploy_changes [23:09:17] (fyi) [23:09:25] Krinkle: awesome. is there documentation on how to do that, btw? i'm trying to learn more about it [23:09:27] got it! [23:09:35] If you have a few minutes, I'd rather you try. [23:09:45] Krinkle: absolutely [23:10:00] in that case, scan through the rest of the page first. [23:10:11] 3Beta-Cluster: HHVM emits logs filling /var/log/upstart/hhvm.log and /var/log/syslog/ filling disk - https://phabricator.wikimedia.org/T71976#799624 (10hashar) [23:10:47] 3Beta-Cluster: hhvm apache fills /var/log/apache2 with access logs - https://phabricator.wikimedia.org/T75262#799625 (10hashar) [23:45:31] 3Beta-Cluster: Thumbnail 404s get cached - https://phabricator.wikimedia.org/T69056#799808 (10Tgr) [23:46:29] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%) [23:46:33] 3Deployment-Systems, Release-Engineering, MediaWiki-Core-Team: Update servers in scap rsync proxy pool - https://phabricator.wikimedia.org/T1342#799820 (10greg) [23:48:20] 3Beta-Cluster: Shinken warnings about free space in deployment-cache-upload02 - https://phabricator.wikimedia.org/T76417#799841 (10yuvipanda) [23:51:40] 3Beta-Cluster: Thumbnail 404s get cached - https://phabricator.wikimedia.org/T69056#799878 (10Tgr)