[00:00:38] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1247872 (10greg) [00:00:48] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1247775 (10greg) [01:00:59] PROBLEM - Puppet failure on deployment-graphoid is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:04:27] (03PS1) 10Dduvall: Script for importing Elasticsearch mappings [integration/raita] - 10https://gerrit.wikimedia.org/r/207695 [01:08:33] lots of beta-mediawiki-config-update-eqiad stuck [01:25:58] RECOVERY - Puppet failure on deployment-graphoid is OK: OK: Less than 1.00% above the threshold [0.0] [01:32:13] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1248038 (10GWicke) An example for why we should have some sanity checks during deploy: https://wikitech.wikimedia.org/wiki/Incident_docume... [01:33:40] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1248043 (10Yurik) VM is created, configured at https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=configure&project=deployment-prep&instanceid=b77ac7a3-11e2-400f-beea-27308bda007f&re... [01:34:14] 10Beta-Cluster, 10Graphoid: Deploy Graphoid on Beta Cluster - https://phabricator.wikimedia.org/T97606#1248044 (10yuvipanda) [02:43:31] (03PS1) 10PleaseStand: Add script that checks for incorrectly capitalized class names [tools/code-utils] - 10https://gerrit.wikimedia.org/r/207720 [03:11:35] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1248170 (10ssastry) Okay, we should also have @catrope, @Jdforrester-WMF,... [03:49:52] RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1086 bytes in 0.029 second response time [04:40:19] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #429: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/429/ [05:06:45] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [06:30:09] Yippee, build fixed! [06:30:09] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #598: FIXED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/598/ [07:14:52] Yippee, build fixed! [07:14:53] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #426: FIXED in 49 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/426/ [08:36:31] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.003 second response time [08:38:27] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.004 second response time [08:42:56] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #585: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/585/ [08:50:58] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.005 second response time [08:52:48] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50725 bytes in 0.012 second response time [08:54:24] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50724 bytes in 0.007 second response time [09:17:42] (03CR) 10Zfilipin: [C: 031] "Looks good to me in general. See one minor inline comment." (031 comment) [selenium] - 10https://gerrit.wikimedia.org/r/207324 (owner: 10Dduvall) [09:21:08] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #413: STILL FAILING in 7.1 sec: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/413/ [10:13:24] 10Browser-Tests: Create new account at Sauce Labs for running Jenkins jobs - https://phabricator.wikimedia.org/T97549#1248571 (10zeljkofilipin) The e-mail is created. [10:13:46] 10Browser-Tests: Create new account at Sauce Labs for running Jenkins jobs - https://phabricator.wikimedia.org/T97549#1248575 (10zeljkofilipin) 5Open>3Resolved [10:13:51] 6Release-Engineering: Convert old wmf/* deployment branches to tags (recurring chore) - https://phabricator.wikimedia.org/T1288#1248576 (10hashar) Maybe the make-wmf-branch could be taught how to convert obsolete branches to tags? This way the conversion will be part of the process and we could close this task. [10:20:04] 10Continuous-Integration-Infrastructure: integration-saltmaster stalled / can not reboot due to labvirt1005 - https://phabricator.wikimedia.org/T97533#1248593 (10hashar) 5Open>3Resolved a:3hashar It is back up. The instance went crazy because of labvirt1005 issue (T97521). [10:38:55] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #303: FAILURE in 2.5 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/303/ [10:45:49] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [43200.0] [10:56:32] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 47079 bytes in 1.622 second response time [10:57:47] (03PS1) 10Pastakhov: fix mwext-PhpTagsFunctions [integration/config] - 10https://gerrit.wikimedia.org/r/207754 [10:57:50] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 28387 bytes in 2.995 second response time [10:59:26] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47368 bytes in 0.645 second response time [11:10:41] (03CR) 1020after4: [C: 032] git-logs: Make log_updates python3 compatible [tools/release] - 10https://gerrit.wikimedia.org/r/181042 (owner: 10Legoktm) [11:10:50] (03Merged) 10jenkins-bot: git-logs: Make log_updates python3 compatible [tools/release] - 10https://gerrit.wikimedia.org/r/181042 (owner: 10Legoktm) [11:10:57] (03CR) 1020after4: [C: 032] git-logs: Port log_updates.py to python3 [tools/release] - 10https://gerrit.wikimedia.org/r/181043 (owner: 10Legoktm) [11:11:04] (03Merged) 10jenkins-bot: git-logs: Port log_updates.py to python3 [tools/release] - 10https://gerrit.wikimedia.org/r/181043 (owner: 10Legoktm) [11:14:36] 6Release-Engineering, 6Engineering-Community, 3ECT-May-2015: Lyon -> Annecy Transportation Info to RelEng Team - https://phabricator.wikimedia.org/T93686#1248705 (10Qgil) [11:21:09] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:27:05] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [11:36:08] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:38:31] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47282 bytes in 2.215 second response time [11:41:05] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 47072 bytes in 5.849 second response time [11:42:51] (03PS2) 1020after4: [tox] Add environment for flake8 under python3 [tools/release] - 10https://gerrit.wikimedia.org/r/181044 (owner: 10Legoktm) [11:43:26] (03CR) 1020after4: [C: 032] [tox] Add environment for flake8 under python3 [tools/release] - 10https://gerrit.wikimedia.org/r/181044 (owner: 10Legoktm) [11:43:42] (03Merged) 10jenkins-bot: [tox] Add environment for flake8 under python3 [tools/release] - 10https://gerrit.wikimedia.org/r/181044 (owner: 10Legoktm) [11:56:28] hashar: looks like there is something wrong with the jenkins irc plugin again :( https://integration.wikimedia.org/ci/view/BrowserTests/view/-All/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/494/console [11:56:43] I have to go to a meeting, I have just noticed it [11:56:54] zeljkof-meetign: dealing with it [11:56:55] https://integration.wikimedia.org/ci/monitoring?part=threadsDump [11:57:03] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:57:04] hashar: thanks [11:59:44] zeljkof-meeting: have you tried changing some config change ? [11:59:47] grr [11:59:56] zeljkof-meeting: have you tried changing the Jenkins configuration recently ? [12:00:10] hashar: yes, I have deleted env variable [12:00:17] working on this https://phabricator.wikimedia.org/T89342 [12:02:02] hashar: so changing jenkins configuration messes up irc plugin? [12:08:10] 10Continuous-Integration-Infrastructure, 7Jenkins, 7Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#1248764 (10hashar) [12:08:58] 10Continuous-Integration-Infrastructure, 7Jenkins, 7Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#1210216 (10hashar) Jenkins deadlocked again. I took a threaddump available at P584. I have updated this ta... [12:09:06] !log restarting Jenkins https://phabricator.wikimedia.org/T96183 [12:09:09] zeljkof-meeting: no idea [12:09:12] Logged the message, Master [12:10:29] (03PS1) 10Florianschmidtwelzow: Add OOJsUIAjaxLogin extension [integration/config] - 10https://gerrit.wikimedia.org/r/207758 [12:11:11] (03PS2) 10Florianschmidtwelzow: Add OOJsUIAjaxLogin extension [integration/config] - 10https://gerrit.wikimedia.org/r/207758 [12:19:48] hashar: meeting canceled, thanks for rebooting jenkins [12:23:07] I am deleting SAUCE_ONDEMAND_ACCESS_KEY env variable again from jenkins, looks like reboot added it back :O [12:25:47] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #496: FAILURE in 2 min 7 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/496/ [12:26:07] Starting build #1 for job hashar-poke-releng [12:26:07] Project hashar-poke-releng build #1: SUCCESS in 0.14 sec: https://integration.wikimedia.org/ci/job/hashar-poke-releng/1/ [12:28:12] Project hashar-poke-releng build #2: SUCCESS in 64 ms: https://integration.wikimedia.org/ci/job/hashar-poke-releng/2/ [12:28:16] Yippee, build fixed! [12:28:17] Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #497: FIXED in 48 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/497/ [12:29:52] (03PS1) 10Zfilipin: Move SAUCE_ONDEMAND_ACCESS_KEY environment variable to Jenkins credential store [integration/config] - 10https://gerrit.wikimedia.org/r/207760 (https://phabricator.wikimedia.org/T89342) [12:30:32] hashar: can you review this? https://gerrit.wikimedia.org/r/#/c/207760/ [12:30:51] all browsertest* jobs will fail until this is merged [12:31:11] I had to remove SAUCE_ONDEMAND_ACCESS_KEY env variable from Jenkins to test if the patch works :) [12:32:14] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #632: ABORTED in 13 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/632/ [12:33:43] (03CR) 10Zfilipin: "Please review and merge as soon as possible. I had to remove SAUCE_ONDEMAND_ACCESS_KEY environment variable from Jenkins to test if this w" [integration/config] - 10https://gerrit.wikimedia.org/r/207760 (https://phabricator.wikimedia.org/T89342) (owner: 10Zfilipin) [12:33:57] zeljkof: just merge it ? [12:34:24] zeljkof: I guess you tested it already havent you? [12:34:25] hashar: sure, can you take a quick look first? [12:34:42] hashar: of course, see the links in gerrit comments [12:35:16] it is a 4 line change, just wanted to avoid reverting if you think it should be done differently [12:35:24] zeljkof: since you confirmed it works, just +2 it :] [12:35:46] hashar: ok, in that case +2ing and updating all jobs [12:36:35] (03CR) 10Zfilipin: [C: 032] "hashar suggested at #wikimedia-releng to just +2 it :)" [integration/config] - 10https://gerrit.wikimedia.org/r/207760 (https://phabricator.wikimedia.org/T89342) (owner: 10Zfilipin) [12:36:49] zeljkof: well since you removed the variable [12:36:52] and updated jobs already [12:37:01] and confirmed it works ... :D [12:37:18] hashar: I have updated just one job, to see if it works [12:37:40] hashar: and the variable is easy to put back, but I would like to avoid that [12:38:27] 10Continuous-Integration-Infrastructure, 7Jenkins, 7Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#1248805 (10hashar) [12:38:38] 10Continuous-Integration-Infrastructure, 7Jenkins, 7Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#1210216 (10hashar) I have enabled some Jenkins logs at https://integration.wikimedia.org/ci/log/IRC%20IM%2... [12:40:05] (03Merged) 10jenkins-bot: Move SAUCE_ONDEMAND_ACCESS_KEY environment variable to Jenkins credential store [integration/config] - 10https://gerrit.wikimedia.org/r/207760 (https://phabricator.wikimedia.org/T89342) (owner: 10Zfilipin) [12:48:52] (03CR) 10Zfilipin: "I have updated all browsertests* jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/207760 (https://phabricator.wikimedia.org/T89342) (owner: 10Zfilipin) [12:48:53] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #240: SUCCESS in 49 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/240/ [12:58:46] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #40: FAILURE in 9.1 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/40/ [13:00:45] Yippee, build fixed! [13:00:46] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #41: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/41/ [13:02:41] 10Browser-Tests: Create new account at Sauce Labs for running Jenkins jobs - https://phabricator.wikimedia.org/T97549#1248848 (10zeljkofilipin) I have updated Jenkins, all browsertests* jobs will now use wikimedia-jenkins Sauce Labs account. [13:06:41] 10Browser-Tests: Create new account at Sauce Labs for running Jenkins jobs - https://phabricator.wikimedia.org/T97549#1248859 (10zeljkofilipin) I have documented username/password at https://office.wikimedia.org/wiki/Selenium_passwords [13:08:41] hashar: do you want to disclose this? https://phabricator.wikimedia.org/T89226 [13:23:42] zeljkof-lunch: I have replied on the task [13:40:22] !log Jenkins: downgrading IRC plugin from 2.26 to 2.25 [13:40:27] Logged the message, Master [13:41:34] 6Release-Engineering, 10Ops-Access-Requests, 6operations: Grant access for aklapper to phab-admins - https://phabricator.wikimedia.org/T97642#1248924 (10chasemp) 3NEW [13:43:32] 10Continuous-Integration-Infrastructure, 7Jenkins, 7Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#1248940 (10hashar) I have downgraded the IRC plugin from 2.26 to 2.25. The upgrade might have caused the i... [13:59:02] 10Continuous-Integration-Infrastructure, 7Jenkins, 7Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#1248967 (10hashar) [[ https://github.com/jenkinsci/ircbot-plugin/commit/98b0105a743d062abf957c285cdded06fe... [14:10:52] 10Continuous-Integration-Infrastructure, 7Jenkins, 7Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#1248988 (10hashar) Filled upstream as https://issues.jenkins-ci.org/browse/JENKINS-28175 [14:17:38] zeljkof-lunch: ok the IRC issue should be gone now [14:17:41] I have downgraded the plugin [14:17:52] !log Jenkins: properly downgraded IRC plugin from 2.26 to 2.25 [14:17:57] Logged the message, Master [14:43:59] hashar: great! [14:46:12] Yippee, build fixed! [14:46:12] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #237: FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/237/ [14:46:39] hashar: line 171 in manifests/role/zuul.pp the bug is closed, should anything be down about it ? [14:46:49] *donw [14:46:50] e [15:23:27] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:26:28] matanya: cant you point to the bug ? :D [15:49:11] 10Browser-Tests, 6Release-Engineering, 10VisualEditor: Selenium bug with Firefox causes VE test failure - https://phabricator.wikimedia.org/T90651#1249211 (10greg) p:5High>3Unbreak! [15:49:14] 10Browser-Tests, 7Puppet: [Regression] QA: Puppet failing for Role::Ci::Slave::Browsertests/elasticsearch - https://phabricator.wikimedia.org/T74255#1249214 (10greg) p:5Low>3Normal [16:02:52] 10Browser-Tests, 6Release-Engineering, 10VisualEditor: Selenium bug with Firefox causes VE test failure - https://phabricator.wikimedia.org/T90651#1249246 (10greg) p:5Unbreak!>3High [16:03:12] 10Browser-Tests, 7Puppet: [Regression] QA: Puppet failing for Role::Ci::Slave::Browsertests/elasticsearch - https://phabricator.wikimedia.org/T74255#1249247 (10greg) p:5Normal>3Low [16:13:41] 10Browser-Tests, 6Release-Engineering, 10MediaWiki-Vagrant, 5Patch-For-Review: Vagrant command for running browser tests - https://phabricator.wikimedia.org/T96283#1249297 (10greg) a:3dduvall [16:29:07] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:29:46] (03CR) 10Dduvall: Raita Elasticsearch logging (031 comment) [selenium] - 10https://gerrit.wikimedia.org/r/207324 (owner: 10Dduvall) [16:30:31] (03PS5) 10Dduvall: Raita Elasticsearch logging [selenium] - 10https://gerrit.wikimedia.org/r/207324 [17:41:55] hoping for some cucumber advice, There is a common pattern used in the cirrus suite, http://pastie.org/10122491, that i think could be done much better so the error message isnt 'expected: true got: false' but i'm not sure how i should rewrite it. What rspec or ruby features should i be considering to fix this up? [17:42:30] ebernhardson: in a pairing session atm but i can take a look in 20 [17:42:35] marxarelli: thanks [17:47:24] ebernhardson: had a pause in the session. i think this should work http://pastie.org/10122500# [17:48:28] O_O that is some literate programmering [17:49:10] wow, if its that simple :) [17:49:52] you could use Then but i personally like to repeat myself a little bit if it makes it more clear [17:50:59] marxarelli: thanks. this pattern is used in 5 or 6 places, i'll try and reapply that kind of solution and come back if more questions. [17:51:21] ebernhardson: no problem! [17:56:37] 10Beta-Cluster, 10Parsoid: Intermittent "Failed contacting Parsoid: There was a problem during the HTTP request: 503 Service Unavailable " when requesting http://en.wikipedia.beta.wmflabs.org/w/api.php - https://phabricator.wikimedia.org/T97491#1249781 (10Mattflaschen) [17:57:23] 10Beta-Cluster, 10Parsoid: Intermittent "Failed contacting Parsoid: There was a problem during the HTTP request: 503 Service Unavailable " when requesting http://en.wikipedia.beta.wmflabs.org/w/api.php - https://phabricator.wikimedia.org/T97491#1249787 (10Catrope) 5Open>3Resolved a:3Catrope Should be fix... [18:05:54] 10Continuous-Integration-Infrastructure, 10Gitblit-Deprecate, 7Technical-Debt: Remove dependency on git.wikimedia.org - https://phabricator.wikimedia.org/T74001#1249840 (10Negative24) [18:07:43] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1249846 (10ssastry) @hashar so, based on IRC discussion, we want beta clu... [18:19:22] 6Release-Engineering, 6operations: Move sudo permissions for deployment from modules/mediawiki/manifests/users.pp to data.yaml - https://phabricator.wikimedia.org/T97678#1249908 (10chasemp) 3NEW [18:23:28] 6Release-Engineering, 6operations: Move sudo permissions for deployment from modules/mediawiki/manifests/users.pp to data.yaml - https://phabricator.wikimedia.org/T97678#1249921 (10chasemp) [18:37:14] 6Release-Engineering, 6operations: Move sudo permissions for deployment from modules/mediawiki/manifests/users.pp to data.yaml - https://phabricator.wikimedia.org/T97678#1249993 (10chasemp) [18:42:32] 6Release-Engineering, 6operations: Move sudo permissions for deployment from modules/mediawiki/manifests/users.pp to data.yaml - https://phabricator.wikimedia.org/T97678#1250005 (10chasemp) https://gerrit.wikimedia.org/r/#/c/207877/ [18:53:37] Yippee, build fixed! [18:53:38] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #304: FIXED in 36 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/304/ [19:00:41] !log Depooled integration-slave-trusty-1013 for labs maintenance (per andrewbogott) [19:00:48] Logged the message, Master [19:04:53] PROBLEM - Host integration-slave-trusty-1013 is DOWN: CRITICAL - Host Unreachable (10.68.18.28) [19:21:47] 10Continuous-Integration-Infrastructure, 10Wikimedia-Fundraising-CiviCRM: Disable job on CRM deployment branch - https://phabricator.wikimedia.org/T94586#1250138 (10awight) p:5Normal>3High Raising the priority, this is now an obstacle to deployment. [19:22:55] RECOVERY - Host integration-slave-trusty-1013 is UP: PING OK - Packet loss = 0%, RTA = 1.89 ms [19:26:39] !log Repooled integration-slave-trusty-1013. IP unchanged. [19:26:43] Logged the message, Master [19:30:27] PROBLEM - Host integration-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.16.42) [19:30:47] RECOVERY - Host integration-puppetmaster is UP: PING OK - Packet loss = 0%, RTA = 0.67 ms [19:34:54] 6Release-Engineering, 6operations, 5Patch-For-Review: Move sudo permissions for deployment from modules/mediawiki/manifests/users.pp to data.yaml - https://phabricator.wikimedia.org/T97678#1250166 (10chasemp) 5Open>3Resolved a:3chasemp [19:36:33] PROBLEM - Puppet failure on integration-raita is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:01:20] hi [20:01:30] beta labs updating is fucked again [20:01:31] http://bits.beta.wmflabs.org/static-master/resources/lib/oojs-ui/oojs-ui-mediawiki-noimages.css [20:01:33] RECOVERY - Puppet failure on integration-raita is OK: OK: Less than 1.00% above the threshold [0.0] [20:01:37] i am getting an ancient version of this file [20:01:50] (if it has "font-size: 0.8em;" in it, it's old) [20:02:37] would be nice if it could be unfucked. how do i help? [20:06:52] matanya: what do you get when using a cache breaker? ie ...css?foo [20:07:03] cause either the varnish cache is stalled [20:07:15] or the css file has not been deployed [20:07:19] MatmaRex: ^^^ [20:07:29] (03PS2) 10Dduvall: Script for managing Elasticsearch mappings [integration/raita] - 10https://gerrit.wikimedia.org/r/207695 [20:07:58] MatmaRex: bah cache is stalled [20:08:06] (03CR) 10Dduvall: [C: 032] Moved index.html to a docroot directory [integration/raita] - 10https://gerrit.wikimedia.org/r/207679 (owner: 10Dduvall) [20:08:07] hasharAway: http://bits.beta.wmflabs.org/static-master/resources/lib/oojs-ui/oojs-ui-mediawiki-noimages.css?foo gets me the right contents [20:08:12] < Age: 103619 [20:08:17] (03Merged) 10jenkins-bot: Moved index.html to a docroot directory [integration/raita] - 10https://gerrit.wikimedia.org/r/207679 (owner: 10Dduvall) [20:08:33] MatmaRex: yeah so if you append a ?foo that bypass the varnish cache [20:08:41] seems bits caches for quite a while [20:08:55] < Cache-Control: max-age=2592000 [20:08:55] < Expires: Fri, 29 May 2015 15:20:51 GMT [20:09:40] MatmaRex: there was a bug for it somewhere [20:09:49] MatmaRex: one will have to find out how we invalidate the bits cache in prod [20:09:56] aand figure out why it does not happen on beta [20:10:00] * MatmaRex weeps [20:19:58] PROBLEM - Host integration-slave-trusty-1021 is DOWN: CRITICAL - Host Unreachable (10.68.16.17) [20:24:21] 10Continuous-Integration-Infrastructure: Convert pool from a few large slaves (4X) to more smaller slaves (1X) - https://phabricator.wikimedia.org/T96629#1250345 (10hashar) [20:24:23] 10Continuous-Integration-Infrastructure, 6Labs: Create an instance image like m1.small with 2 CPUs and 30GB space - https://phabricator.wikimedia.org/T96706#1250343 (10hashar) 5Open>3Resolved Resized by @andrew integration-slave-trusty-1021 https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000be1.eqia... [20:24:29] 10Continuous-Integration-Infrastructure, 6Labs: Create an instance image like m1.small with 2 CPUs and 30GB space - https://phabricator.wikimedia.org/T96706#1250346 (10Andrew) ok, I deleted and recreated with 40G. Are the other stats still correct? [20:29:28] hashar: prod does not invalidate cache, resourceloader is too complex to just invaldite random files directly in varnish [20:29:40] the problem is not what prod does that beta doesn't, but what beta is doing that it shouldn't. [20:29:48] It has additional cache headers that should not exist [20:29:51] for static files. [20:31:25] :((( [20:33:56] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:37:28] RECOVERY - Host integration-slave-trusty-1021 is UP: PING OK - Packet loss = 0%, RTA = 0.93 ms [20:58:56] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:02:24] PROBLEM - Puppet failure on integration-slave-trusty-1021 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:17:26] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [21:22:24] RECOVERY - Puppet failure on integration-slave-trusty-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [23:31:27] (03PS1) 10Dduvall: Fix db URL and riot source in index.html [integration/raita] - 10https://gerrit.wikimedia.org/r/208027 [23:32:21] (03CR) 10Dduvall: [C: 032] Script for managing Elasticsearch mappings [integration/raita] - 10https://gerrit.wikimedia.org/r/207695 (owner: 10Dduvall) [23:32:32] (03Merged) 10jenkins-bot: Script for managing Elasticsearch mappings [integration/raita] - 10https://gerrit.wikimedia.org/r/207695 (owner: 10Dduvall) [23:32:46] (03CR) 10Dduvall: [C: 032] Fix db URL and riot source in index.html [integration/raita] - 10https://gerrit.wikimedia.org/r/208027 (owner: 10Dduvall) [23:32:56] (03Merged) 10jenkins-bot: Fix db URL and riot source in index.html [integration/raita] - 10https://gerrit.wikimedia.org/r/208027 (owner: 10Dduvall) [23:45:46] marxarelli: Remember that our privacy policy forbids third-party traffic dependencies [23:46:00] So bootstrap/jquery etc. will have to be added to the repo. [23:46:18] Or use tool-labs cdnjs [23:46:22] (though not for prod) [23:47:08] Krinkle: i just noticed http://performance.wikimedia.org/ has google fonts loading, should we bug ori about that? [23:47:16] see also https://phabricator.wikimedia.org/T96499 [23:47:26] Yup. [23:48:01] * greg-g has to run, will do later if no one beats me to it [23:48:08] Krinkle: ok, got it. what's preferred, cdn or local? [23:48:22] marxarelli: In this case, local will be easiest [23:50:38] Krinkle: cool. do you recommend npm/browserify or bower or? [23:50:58] marxarelli: Neither. [23:51:16] Unless the entire app is written with require() or something like that [23:52:09] Krinkle: so just import them and serve them out of a vendor or similar directory? [23:53:17] Yeah [23:53:20] (i haven't done many purely js apps, if that shows :) [23:53:23] PROBLEM - Puppet failure on integration-slave-trusty-1021 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:53:35] marxarelli: e.g. {docroot}/lib/jquery/jquery-1.9.1.js [23:53:44] download the zip file from getbootstrap.com [23:54:24] The dist one (First one) that is [23:54:49] Krinkle: i noticed integration.wikimedia.org/zuul loading jquery off of bits. couldn't i just do that? [23:54:54] for jquery at least [23:55:17] It shouldn't be doing that [23:55:23] for bootstrap I fixed that [23:55:46] bits is for ResourceLoader. Especially for CSS, loading that out of context doesn't work well. [23:55:51] As well with path mapping etc. [23:56:26] alrighty. i'll stick to local [23:56:55] And more important, it would create a dependency on MediaWiki's version of whatever library you use [23:57:17] Krinkle: right. makes sense