[00:01:10] Ho-hum. https://integration.wikimedia.org/ci/view/Beta/ shows that deployment-bastion.eqiad is idle, but beta-scap-eqiad is waiting for it to have a slot free. [00:01:26] What's the thing we do here? It's been weeks and I've forgotten. [00:06:30] disconnect/reconnect? [00:07:24] * greg-g does that [00:07:56] !log deployment-bastion is idle, yet we have 3 pending jobs waiting for an executer on it - will disconnect/reconnect it in Jenkins [00:07:59] Logged the message, Master [00:10:11] !log after disconnecting, marking temp offline, bringing back online, and launching slave agent: "Slave successfully connected and online" [00:10:14] Logged the message, Master [00:10:41] still nothing... [00:11:16] !log still nothing... [00:11:19] Logged the message, Master [00:12:04] 10Continuous-Integration-Infrastructure, 10Gather, 5Patch-For-Review: Set up qunit Jenkins job for Extension:Gather - https://phabricator.wikimedia.org/T91708#1240549 (10Jdlrobson) @legoktm the failure is due to T97268 [00:17:03] !log after the 3rd or so time doing it (while on the Golden Gate Bridge, btw) it worked [00:17:07] Logged the message, Master [00:17:22] James_F: ^ [00:21:41] PROBLEM - App Server Main HTTP Response on deployment-mediawiki01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:08] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:18] greg-g: Thanks. [00:23:59] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:06] greg-g: Hah: https://integration.wikimedia.org/ci/view/Beta/jenkins100k/ [00:25:34] legoktm: should be safe to return whatever you like ;) [00:25:41] :D [00:26:33] RECOVERY - App Server Main HTTP Response on deployment-mediawiki01 is OK: HTTP OK: HTTP/1.1 200 OK - 47082 bytes in 1.770 second response time [00:27:34] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47285 bytes in 3.696 second response time [00:28:48] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 28598 bytes in 0.750 second response time [01:24:36] RECOVERY - Puppet failure on integration-raita is OK: OK: Less than 1.00% above the threshold [0.0] [01:36:39] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<30.00%) [03:04:23] Project beta-scap-eqiad build #50727: FAILURE in 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50727/ [03:14:57] Yippee, build fixed! [03:14:58] Project beta-scap-eqiad build #50728: FIXED in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50728/ [04:53:25] 10Beta-Cluster, 10MediaWiki-extensions-CentralAuth: CentralAuth on beta throws "Cannot access the database: Unknown database 'dawiki'" when user is not found - https://phabricator.wikimedia.org/T97388#1240778 (10Krinkle) 3NEW [05:06:46] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [05:07:29] 10Beta-Cluster, 10MediaWiki-extensions-CentralAuth: CentralAuth on beta throws "Cannot access the database: Unknown database 'dawiki'" when user is not found - https://phabricator.wikimedia.org/T97388#1240806 (10Krinkle) The same happens when trying to create a new account via e.g. http://deployment.wikimedia.... [05:08:22] 10Beta-Cluster, 10MediaWiki-extensions-CentralAuth: CentralAuth on beta throws "Cannot access the database: Unknown database 'dawiki'" for new users - https://phabricator.wikimedia.org/T97388#1240807 (10Krinkle) [05:31:42] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #53: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/53/ [06:36:37] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [07:29:43] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce build #20: FAILURE in 20 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-10-sauce/20/ [07:45:28] 10Browser-Tests, 3Gather Sprint Forward, 6Mobile-Web, 10Mobile-Web-Sprint-45-Snakes-On-A-Plane, 5Patch-For-Review: Fix failed MobileFrontend browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94156#1241002 (10zeljkofilipin) Wait, not all MobileFrontend Jenkins jobs are fixed: https://integra... [08:05:26] YuviPanda: did you say on loomio that you can help with irccloud? [09:02:10] 10Beta-Cluster, 10MediaWiki-extensions-CentralAuth: CentralAuth on beta throws "Cannot access the database: Unknown database 'dawiki'" for new users - https://phabricator.wikimedia.org/T97388#1241060 (10hashar) That is because a bunch of databases have been added to the mediawiki-config but have not been prope... [09:02:27] 10Beta-Cluster, 10MediaWiki-extensions-CentralAuth: CentralAuth on beta throws "Cannot access the database: Unknown database 'dawiki'" for new users - https://phabricator.wikimedia.org/T97388#1241062 (10hashar) [09:02:30] 10Beta-Cluster, 7Blocked-on-RelEng, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, and 3 others: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1064892 (10hashar) [09:03:04] 10Beta-Cluster, 7Blocked-on-RelEng, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, and 3 others: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1064892 (10hashar) @KartikMistry can you take care of finishing the databases ins... [09:12:25] 10Continuous-Integration-Infrastructure, 10Gather, 5Patch-For-Review: Set up qunit Jenkins job for Extension:Gather - https://phabricator.wikimedia.org/T91708#1241073 (10hashar) [09:14:23] Project beta-scap-eqiad build #50764: FAILURE in 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50764/ [09:30:11] hashar: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Add_a_wiki#Step_2:_update_the_Apache_configuration - seems outdated? [09:30:33] hashar: there is no /a/common now [09:31:12] 10Beta-Cluster, 7Blocked-on-RelEng, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, and 3 others: Setup new wikis in Beta Cluster for Content Translation - https://phabricator.wikimedia.org/T90683#1241086 (10KartikMistry) @hashar https://wikitech.wikimedia.org/wiki/Nova_Resourc... [09:31:29] ok. better to comment on task :) [09:32:08] kart_: seems the doc hasn't been updated in a while :/ [09:32:39] kart_: I think the apache conf can be skilled [09:32:55] you still need to use the addWiki.php script though which takes care of creating the database [09:35:18] Yippee, build fixed! [09:35:19] Project beta-scap-eqiad build #50766: FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50766/ [09:38:09] hashar: sure. I'm getting this, DB connection error: Unknown database 'dawiki' (10.68.16.193) [11:06:00] 10Continuous-Integration-Infrastructure: Use systemd for xvfb service on Debian/Jessie - https://phabricator.wikimedia.org/T95003#1241252 (10hashar) [11:06:54] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Isolation, 6operations, 7Nodepool: Use systemd for Nodepool - https://phabricator.wikimedia.org/T96867#1241257 (10hashar) From T95003 @joe pointed to our puppet define `base::service_unit` which maybe a good template to add systemd to nodepoo... [11:10:46] 10Continuous-Integration-Infrastructure: Use systemd for xvfb service on Debian/Jessie - https://phabricator.wikimedia.org/T95003#1241272 (10hashar) [11:10:50] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Isolation, 6operations, 7Nodepool: Use systemd for Nodepool - https://phabricator.wikimedia.org/T96867#1241273 (10hashar) [11:18:41] 10Continuous-Integration-Infrastructure, 7Zuul: Zuul status page should show the pipelines "window" value - https://phabricator.wikimedia.org/T93701#1241299 (10hashar) p:5Normal>3Low Now that our Zuul status page is closer to the upstream page, I agree that should be proposed upstream. Not a priority though. [11:38:03] 10Deployment-Systems, 6Release-Engineering, 7Epic: Rethinking our deployment process - https://phabricator.wikimedia.org/T89945#1241313 (10Yurik) Lets call this PHP deployment process. Our services deployment may also need rethinking - it takes nearly forever to deploy any new service -- whereas as we remem... [12:39:14] 10Browser-Tests, 6Release-Engineering, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1241421 (10WMDE-Fisch) I made some progress in analysing the problem. It seems to occur because the browser window on... [13:08:46] (03CR) 10Hashar: [C: 032] Config for integration/raita [integration/config] - 10https://gerrit.wikimedia.org/r/206967 (owner: 10Dduvall) [13:10:22] (03Merged) 10jenkins-bot: Config for integration/raita [integration/config] - 10https://gerrit.wikimedia.org/r/206967 (owner: 10Dduvall) [13:10:33] (03CR) 10Hashar: "You will want to integration the jshint and jsonlint jobs in the npm test entry point as described at https://www.mediawiki.org/wiki/Conti" [integration/config] - 10https://gerrit.wikimedia.org/r/206967 (owner: 10Dduvall) [13:34:49] how I ssh into gallium? its having trouble building my stuff and I want to clear its maven cache for lucene 4.9.0 [13:34:55] https://integration.wikimedia.org/ci/job/search-highlighter/249/consoleFull [13:42:56] hashar: can you help me to debug a build issue? ^^^^ [13:52:36] manybubbles: maybe :) [13:52:44] manybubbles: I have a meeting in a few minutes though [13:53:40] maybe the maven cache is borked? /var/lib/jenkins-slave/.m2/repository [13:55:22] hashar: how can I ssh into gallium in the first place? is it gallium.eqiad.wmflabs? wmnet? neither work for me [13:57:02] hashar: is the CI meeting here? [13:57:08] or in #wikimedia-office? [13:57:24] the calendar says here, but I thing it was in office channel before [13:58:11] 10Beta-Cluster, 10MediaWiki-extensions-GWToolset, 6Multimedia: GWToolset XML upload fails with “The file that was uploaded exceeds the upload_max_filesize and/or the post_max_size directive in php.ini” on Commons Beta - https://phabricator.wikimedia.org/T97415#1241568 (10JeanFred) 3NEW [14:00:35] zeljkof: #wikimedia-office [14:00:45] hashar: joining, thanks [14:00:51] hashar: ignore, I figure out my problem: https://gerrit.wikimedia.org/r/#/c/207072 [14:01:48] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/206967/ [14:01:52] Logged the message, Master [14:08:50] manybubbles: ah good to know! [14:09:17] pom is da beast https://gerrit.wikimedia.org/r/#/c/207072/1..2/pom.xml,unified :] [14:10:21] (03PS1) 10Dduvall: Enforce jshint linting [integration/raita] - 10https://gerrit.wikimedia.org/r/207103 [14:11:57] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: LocalSettings.php should be copied in teardown instead of setup - https://phabricator.wikimedia.org/T90613#1241607 (10Krinkle) 5Open>3Resolved a:3Krinkle [14:13:58] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Disable xdebug's html formatting of PHP errors for Apache on Jenkins slaves - https://phabricator.wikimedia.org/T97040#1241611 (10JanZerebecki) p:5Triage>3Low [14:15:37] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Have jenkins jobs logrotate their build history - https://phabricator.wikimedia.org/T91396#1241614 (10hashar) Update as of Apr 28th ``` $ ssh gallium.wikimedia.org 'grep -L daysToKeep /var/lib/jenkins/jobs/*/config.xml|cut -d/ -f6' analytics-libanon a... [14:23:48] 10Continuous-Integration-Infrastructure: Update jobs to use zuul-cloner with git cache - https://phabricator.wikimedia.org/T97098#1241621 (10Krinkle) [14:26:31] 10Continuous-Integration-Infrastructure, 7Zuul: Zuul-cloner should use hard links when fetching from cache-dir - https://phabricator.wikimedia.org/T97106#1241629 (10hashar) a:3hashar Our Zuul .deb packages need to incorporate https://review.openstack.org/#/c/117626/4 I will handle the patch management. [14:28:47] 10Continuous-Integration-Infrastructure, 7Zuul: Zuul-cloner should use hard links when fetching from cache-dir - https://phabricator.wikimedia.org/T97106#1241637 (10hashar) p:5Triage>3High [14:30:28] hashar: sawa talk on http://cask.co/product/coopr/ last night, just a cool thing [14:30:44] not suggesting we adopt it but [14:30:54] if we were to do our own thing I would steal half of these ideas in a heartbeat [14:33:24] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-SpamBlacklist: Figure out a system to override default settings when in test context - https://phabricator.wikimedia.org/T89096#1241661 (10JanZerebecki) Do you have an idea how to solve the ConfirmEdit example for when it is installed and the inte... [14:34:52] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-SpamBlacklist: Figure out a system to override default settings when in test context - https://phabricator.wikimedia.org/T89096#1241666 (10JanZerebecki) [14:36:59] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #102: FAILURE in 8 min 58 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/102/ [14:37:47] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-SpamBlacklist: Figure out a system to override default settings when in test context - https://phabricator.wikimedia.org/T89096#1241680 (10JanZerebecki) p:5Triage>3High [14:42:57] PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:35] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:34] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47275 bytes in 8.154 second response time [14:49:46] chasemp: hello! Seems like an utility to easily spawn a cluster of applications right? Ubuntu has something similar named Juju http://www.ubuntu.com/cloud/tools/juju [14:50:25] chasemp: in theory you could spawn a mediawiki cluster with varnishes/memcached/database etc [14:55:34] oh juju, I have good buddies who work on the Juju UI team [14:57:50] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1241698 (10Krinkle) [14:59:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:05] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1241700 (10hashar) >>! In T92871#1231878, @Mattflaschen wrote: > I don't... [15:01:30] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1241701 (10hashar) a:3hashar Investigating [15:04:34] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 47265 bytes in 7.939 second response time [15:05:17] 10Continuous-Integration-Infrastructure: reduce copies of mediawiki/core in workspaces - https://phabricator.wikimedia.org/T93703#1241708 (10Krinkle) [15:05:19] 10Continuous-Integration-Infrastructure: Jenkins jobs must wipe workspace - https://phabricator.wikimedia.org/T96627#1241709 (10Krinkle) [15:07:50] RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1086 bytes in 1.539 second response time [15:14:01] PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:38] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:18:57] RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1086 bytes in 5.635 second response time [15:23:27] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:30:14] mh between today and yesterday which jobs zuul triggered for Wikidata changed. but the config in git didn't change. anyone knows what happened? https://gerrit.wikimedia.org/r/#/c/206770/ -> https://gerrit.wikimedia.org/r/#/c/207055/ phplint is missing [15:33:40] (03PS1) 10JanZerebecki: Trigger phplint on Wikidata again [integration/config] - 10https://gerrit.wikimedia.org/r/207122 [15:34:18] (03CR) 10jenkins-bot: [V: 04-1] Trigger phplint on Wikidata again [integration/config] - 10https://gerrit.wikimedia.org/r/207122 (owner: 10JanZerebecki) [15:35:05] Project beta-scap-eqiad build #50801: FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50801/ [15:35:30] :( [15:36:37] 'nother day, 'nother new beta-scap-eqiad failure :( [15:36:48] CalledProcessError: Command 'cp -r "/tmp/scap_l10n_3032739272"/* "/srv/mediawiki-staging/php-master/cache/l10n"' returned non-zero exit status 1 [15:37:02] right after 15:35:05 15:35:05 Finished mw-update-l10n (duration: 10m 57s) [15:37:18] probably file permissions [15:37:31] * bd808 shakes fist at puppet [15:37:38] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace.root.byte_percentfree (<10.00%) WARN: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<100.00%) [15:37:49] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1241795 (10hashar) Jenkins job beta-parsoid-update-eqiad is triggered on... [15:38:02] or that [15:38:55] :( [15:38:55] why did all of our free space disappear recently? [15:38:55] 228K free on / [15:38:55] because copy is failing and it doesn't clean up [15:38:55] tmp [15:38:55] wtf [15:39:11] bd808: we should merge that scap patch [15:39:21] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1241798 (10hashar) a:5hashar>3None [15:39:21] *nod* [15:39:35] you wanna? I think I wrote most of the code there now :) [15:39:55] yeah, at least that'll stop the free space being overloaded [15:39:59] ...hopefully [15:40:08] that twentyafterfour is the master of the +1 when you really wanted a +2 :) [15:40:48] is scap live deployment a manual thing? [15:41:01] I mean, deploying updates to scap on tin? [15:41:04] yeah. trebuchet update and push [15:41:19] where was a +2 wanted? [15:41:20] k, wanted to make sure before I merged [15:41:25] *nod* [15:41:34] (03CR) 10Thcipriani: [C: 032] Make scap localization cache build $TMPDIR aware [tools/scap] - 10https://gerrit.wikimedia.org/r/206856 (https://phabricator.wikimedia.org/T97257) (owner: 10Thcipriani) [15:42:38] !log Freed 5G on deployment-bastion by deleting abandoned /tmp/scap_l10n_* directories [15:42:41] Logged the message, Master [15:43:43] I've had people tell me off too many times for giving a +2 without deploying the associated merge that it makes me not want to +2 anything ... apparently +2 equals a level of responsibility that doesn't seem appropriate to me ... [15:44:18] (03Merged) 10jenkins-bot: Make scap localization cache build $TMPDIR aware [tools/scap] - 10https://gerrit.wikimedia.org/r/206856 (https://phabricator.wikimedia.org/T97257) (owner: 10Thcipriani) [15:44:19] (that is, I really don't think +2 should merge, nor imply anything more than approval of the change) [15:44:36] twentyafterfour: *nod* +2 in mediawiki-config assumes immediate deploy n tin. elsewhere that is not the case [15:44:55] <^d> and puppet [15:44:57] bd808: it also does in operations/puppet [15:45:19] sure, but that requires root and well ... none of us have that [15:46:23] bd808: if +2 would not merge then what is the difference to +1 ? [15:46:33] kk, that scap change is on deployment-bastion [15:46:33] #REDIRECT twentyafterfour [15:46:45] oops thx [15:46:48] :) [15:47:04] it should allow the original author to merge [15:47:13] twentyafterfour is more used to a phab workflow where code review and merge are separate steps [15:47:35] bd808: I'm more used to gerrit now, I just don't like it. [15:47:51] * bd808 adds twentyafterfour to the club [15:48:01] gerrit supports that, too [15:48:29] it's been a long time (2+ years) since I did much phabricator workflow and at deviantart we did not use differential (they adopted it after I left) ... if anything I'm more used to the commit first ask questions later workflow [15:48:44] The next battle will be retraining all the people who think the way gerrit is configured at WMF is the only way to do code review since they haven't really ever used anything else [15:49:00] I'm so looking forward to that one [15:49:08] it will be fun times [15:49:09] and by so looking forward, I mean, ughhhhh [15:49:18] "But all code review tools must..." [15:49:20] phabricator is a bit malleable [15:49:25] i think operations/puppet even has gerrit set up like that, but afaik it is still convention to submit ( = merge) directly after a +2 [15:50:01] jzerebecki: you don't need anyone else's reviews to self-merge in ops/puppet [15:50:21] an opsen can submit patch, +2, deploy, all without anyone else's involvment [15:50:35] 10Beta-Cluster, 6Release-Engineering, 10Continuous-Integration-Config, 10Parsoid: Parsoid patches don't update Beta Cluster automatically -- only deploy repo patches seem to update that code - https://phabricator.wikimedia.org/T92871#1241831 (10ssastry) >>! In T92871#1241795, @hashar wrote: > Jenkins job b... [15:50:47] I think jzerebecki is right that zuul/jenkins/whatever doesn't merge there [15:51:00] you have to hit the review and submit button or whatever it's labeld [15:51:01] 10Beta-Cluster, 10Parsoid: Parsoid on beta cluster times out doing API requests - https://phabricator.wikimedia.org/T97421#1241834 (10hashar) 3NEW [15:51:04] greg-g: that is also true for core [15:51:15] at least technically [15:51:20] right [15:51:24] it's actually true for anyone in the wmf ldap group [15:51:27] anyone with +2 can do whatever they want [15:51:33] which is kinda dumb [15:51:41] and freaked gnubeard out when he heard it [15:51:49] not really. it's all social constructs [15:52:11] the part that's dumb/freaked out gnubeard was the everyone in wmf ldap having +2 on core [15:52:13] I seriously think there should be something between +1 and "let's merge it now plz kthx" [15:52:14] waiting for a "change review board member" to fix prod is the word things ever [15:52:22] *worst [15:52:35] also if two people +1 it should equal a +2 ... [15:52:54] I agree, there should just be a "wmfengineer ldap" that has +2, srodlund doesn't need +2 in core [15:52:58] 10Beta-Cluster, 10Parsoid: Parsoid on beta cluster times out doing API requests - https://phabricator.wikimedia.org/T97421#1241853 (10hashar) A curl query works just fine right now: `curl http://wikidata.beta.wmflabs.org/w/api.php` [15:52:59] at least of two people who _could_ +2 it give a +1 it should equal a +2 [15:53:20] Strict 2nd party review workflow is what leads to uncommitted changes in prod. I've seen it over and over [15:54:00] also, *I* got +2 on core on day one even though no one really tested my dev skills in interviews (luckily) [15:54:08] the situation we have now leads to tons of unapproved patches sitting in gerrit which I eventually abandon out of spite or pure apathy [15:54:20] https://old-bugzilla.wikimedia.org/show_bug.cgi?id=60412 [15:54:23] which isn't an attitude I think I should have about my work but it ends up happening [15:54:40] bd808: that's fair but some things are reviewable and some are not out of fairness, changing a dns name or something ok sure, but any configuration change should be reviewed [15:54:50] we just have weird and bad review culture here [15:55:22] SEE! this is going to be such a fun conversation :) [15:55:32] I'm very much for review I think we have a strange combination of too strict and too lax policies [15:56:29] having come from a culture of good review it's hard to put a nametag on it, but if it's not reflexive and expected it seems to be a hinderance and failure [15:56:39] right now a prod hack doesn't get the "wtf did you do?" looks that it should [15:56:42] for mediawiki/core we used to have a hundred+ of people being able to svn commit [15:56:45] and that's basically the story [15:56:52] so post commit review took a while [15:57:05] when we switched to Gerrit the +2 people ended up being only a dozen or so people [15:57:16] so we only reviewed what was finished / ready for merge [15:58:10] twentyafterfour: at least of two people who _could_ +2 it give a +1 it should equal a +2....it took me awhile to figure out this isn't true [15:58:12] Yippee, build fixed! [15:58:12] Project beta-scap-eqiad build #50803: FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50803/ [15:58:13] and I still don't get it [15:58:46] cause +1 folks are just expressing an opinion [15:58:52] but they have no right to get the change to land in [15:58:59] basically +1 isn't work anything and is pointless [15:59:02] sure but...why call it +1 if two of them do not equal 2? [15:59:03] +1 really means nothing [15:59:05] the vote scores are definitely confusing since they dont sum up [15:59:15] so +1 is more like: ho yeah looks nice, I would buy that [15:59:21] I get it, it's just useless [15:59:22] *+1 isn't worth anything* [15:59:27] and +2 is : ok lets send that product to the shop. [16:00:15] it's crazy here, I spent a bit of time trying to get people to agree to a standard when I joined and didn't come up with anything [16:00:43] this is where some of the feel goodery falls down flat and we need someone to say "shut up and do it correctly" [16:00:49] but that is just like an opinion man [16:01:15] +1 is like a thumbsup on facebook. +2 is http://media.giphy.com/media/143vPc6b08locw/giphy.gif [16:01:16] If we had any confidence in our test coverage things would be much easier. As it is a +2 pretty much means "I have thought about all of the potential consequences of this change and am willing to be yelled at if it causes a problems" [16:01:45] code reveiw always means everyone is equally responsible, source and reviewer yes [16:01:59] but if responsible only means blame that a bigger issue than review [16:02:40] many patches are submitted by people with no access to production logs much less shell to fix things. [16:02:52] This puts the burden on the +2'er [16:03:00] because we are crazy [16:03:31] see also the small number of people who are listed as responding on most incident reports [16:03:46] on the first is it different than say a new guy getting their stuff reviewed by a sr? taht's not equally shared responsibility anywhere I've been [16:03:57] I don't necessarily see the parallel [16:04:02] bystander effect here is strong [16:04:40] and the downside of getting involved seems smaller than the upside of getting involved afaict [16:04:49] chasemp: agreed [16:05:33] By leaving roles open and fluid we create a vacuum of responsibility [16:06:17] it's interesting to me that we talked a lot about review and code quality and what makes a sr person [16:06:21] but none of the goals for anyone at all [16:06:27] have code review in them taht I have seen [16:06:54] (at the all staff I mean) [16:07:19] there are 3 people that I go to for +2 in mediawiki-core. that's pretty pathetic [16:07:38] I can think of 2 more that I could probably call on in a pinch [16:07:43] chasemp: the classes for defining goals for c-level personal i have seen teach it this way: define your goals as things that you already reached last year [16:08:12] that is poignant and yeah [16:10:07] bd808: I am not sure who has the bandwith to review mw/core patches nowadays [16:10:20] I myself used to do a lot of review but now focusing on ci / deployment [16:10:21] :( [16:10:47] Aaron, Brad and Kunal make time to do a lot [16:11:19] I wouldn't be surprised to find that the 3 of them +2 >80% of core changes [16:11:23] the rest of mw/core reviewers I guess we are too busy in some other contexts [16:11:54] the weird part about it is you are only too busy if you feel it's not literally your job to do it [16:12:07] as in, can a dev say they didn't meet their goals because of overwhelming code review needs? [16:12:14] idk but I would guess no one else does either [16:12:18] which is sadly the case. Hard to put a KPI on code review [16:12:43] and when you are already asked to do 150% of a normal human's workload ... [16:12:45] yeah it has no immediate tangible feedback [16:13:05] <^d> bd808: +me :) [16:13:20] ^d: yes. you are awesome too :) [16:13:34] bd808: Gerrit review notes would tell [16:13:46] * bd808 stops whining and goes to find work to do [16:13:53] bd808: git fetch gerrit refs/notes/review:refs/notes/review && git log --show-notes=review --since="6 months ago"|grep Code-Review [16:13:53] <^d> Who needs friends when you can spend a saturday on gerrit? :p [16:14:00] heh sorry bd808 for jumping in on your convo [16:14:04] I guess I cannot help myself [16:14:52] bd808: https://phabricator.wikimedia.org/P566 :) [16:16:17] I bet most of those by me were on the far end of 6 months ago [16:17:28] and if you look at that list [16:17:33] most are from the old mediawiki/core team [16:17:40] Umherirrender being a volunteer [16:17:43] ori because ori [16:18:06] and from the other devs teams that is solely Gilles and Matt flaschen [16:18:22] I guess because most devs are already busy enough with their extensions [16:18:29] to be fair one would need to substract the self merges from the count ;) [16:20:46] hasharMeeting: http://koti.kapsi.fi/~federico/crstats/core.txt [16:20:58] 10Continuous-Integration-Infrastructure, 7Upstream, 7Zuul: Zuul Status API cached too long by Varnish - https://phabricator.wikimedia.org/T94796#1241967 (10Krinkle) 5Open>3stalled [16:21:43] legoktm: wow, that's a lot easier than parsing wmbot logs :P [16:21:54] hasharMeeting: ^ [16:22:00] :P [16:22:56] There's also https://github.com/wikimedia/mediawiki/graphs/contributors for commit authors which is quite flexible and sweet [16:23:22] https://github.com/wikimedia/mediawiki/graphs/contributors?from=2014-08-17 [16:23:27] and http://korma.wmflabs.org/browser/scr.html [16:24:24] Though beware that github's graphs only includes commit authors that are confirmed with a github account [16:24:26] !log Updated scap to ef15380 (Make scap localization cache build $TMPDIR aware) [16:24:30] So many awesome folks aren't in there [16:24:32] Logged the message, Master [16:25:03] You can link multiple address to one github account though [16:26:15] you can also link your @users.mediawiki.org email from SVN even though you can't confirm it [16:26:33] (03PS1) 10Legoktm: Use generic phpunit job for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/207132 [16:26:52] (03PS2) 10Legoktm: Use generic phpunit job for extensions with dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/207132 (https://phabricator.wikimedia.org/T96690) [16:27:13] Indeed [16:27:35] Oh, I guess I did that. [16:27:44] Reedy too [16:28:08] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 6Scrum-of-Scrums, 6operations, and 2 others: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#1242001 (10greg) 5Open>3stalled [16:28:17] <^d> Reason #85 I want to rewrite our core history again [16:28:26] <^d> Those users.mw.o addresses were stupid [16:29:05] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:34:39] ^d: and I thought you had an alias file to properly map svn users to email/names [16:34:45] it is not that much of an issue anyway [16:35:01] I dont see us rewriting the history since we refer to commit sha1 everywhere in the commit messages or in bugs [16:35:17] * ^d shrugs [16:35:19] <^d> We could fork :p [16:35:40] ahah [16:40:05] (03PS1) 10Legoktm: Run mediawiki/core phpunit tests on sqlite again (in addition to mysql) [integration/config] - 10https://gerrit.wikimedia.org/r/207135 [16:45:09] (03CR) 10JanZerebecki: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/207122 (owner: 10JanZerebecki) [16:51:19] (03CR) 10Hashar: [C: 04-1] Trigger phplint on Wikidata again (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/207122 (owner: 10JanZerebecki) [16:51:29] jzerebecki: yaml typo :) [16:52:16] hasharMeeting: gah [16:52:19] thx [16:57:11] hasharMeeting: can you explain to me why https://gerrit.wikimedia.org/r/#/c/207055/ did not run phplint even thought it is in project-templates:extension-unittests which is enabled for that repo? it worked yesterday. [16:58:11] jzerebecki: i think the job only run when a php file is changed [16:58:19] in zuul layout there should be something like: [16:58:21] job: phplint [16:58:25] files: *.php [16:58:56] jzerebecki: https://github.com/wikimedia/integration-config/blob/master/zuul/layout.yaml#L556-L558 [16:59:27] hasharMeeting: yea that is it. thx [17:00:27] well I am off after the meeting [17:00:43] (03Abandoned) 10JanZerebecki: Trigger phplint on Wikidata again [integration/config] - 10https://gerrit.wikimedia.org/r/207122 (owner: 10JanZerebecki) [17:11:48] 6Release-Engineering, 10MediaWiki-Debug-Logging, 6Reading-Infrastructure-Team, 10Wikimedia-Logstash, 7HHVM: Log php fatals with full backtraces again (fatal.log on fluorine) - https://phabricator.wikimedia.org/T89169#1242134 (10bd808) [17:23:00] PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:51] RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1086 bytes in 0.142 second response time [17:48:30] !log Restarting grrrit-wm for config change. [17:48:32] Logged the message, Master [17:49:26] Hmm. [18:01:29] 10Deployment-Systems, 10MediaWiki-ResourceLoader, 6operations: Bad cache stuck due to race condition with scap between different web servers - https://phabricator.wikimedia.org/T47877#1242508 (10Krinkle) [18:06:48] 6Release-Engineering, 10MediaWiki-Installer: Installer should set $wgResourceBasePath before embedding in $wgRightsIcon - https://phabricator.wikimedia.org/T75031#1242537 (10Krinkle) [18:08:51] 6Release-Engineering, 10MediaWiki-Installer: Installer should set $wgResourceBasePath before embedding in $wgRightsIcon - https://phabricator.wikimedia.org/T75031#1242555 (10Krinkle) At run time, in Setup.php, `wgResourceBasePath` defaults to `wgScriptPath`. However, unlike most references to this variable, i... [18:09:04] 6Release-Engineering, 10MediaWiki-Installer: Installer should set $wgResourceBasePath before embedding in $wgRightsIcon - https://phabricator.wikimedia.org/T75031#1242560 (10Krinkle) p:5Low>3Normal [18:21:16] PROBLEM - Puppet failure on integration-zuul-server is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:21:30] PROBLEM - Puppet failure on integration-zuul-packaged is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:21:49] PROBLEM - Puppet failure on integration-puppetmaster is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:25:36] PROBLEM - Puppet failure on integration-raita is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:37:39] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<30.00%) [18:42:17] 10Beta-Cluster: Can't connect to Beta Cluster database deployment-db1 or deployment-db2 (MariaDB down) - https://phabricator.wikimedia.org/T96905#1242741 (10Ryasmeen) This issue is still occurring . Any update on this? [18:50:34] RECOVERY - Puppet failure on integration-raita is OK: OK: Less than 1.00% above the threshold [0.0] [18:51:16] RECOVERY - Puppet failure on integration-zuul-server is OK: OK: Less than 1.00% above the threshold [0.0] [18:51:48] RECOVERY - Puppet failure on integration-puppetmaster is OK: OK: Less than 1.00% above the threshold [0.0] [18:55:35] 10Beta-Cluster: Can't connect to Beta Cluster database deployment-db1 or deployment-db2 (MariaDB down) - https://phabricator.wikimedia.org/T96905#1242819 (10Krenair) Not occurring for me. [19:08:42] 10Beta-Cluster: Can't connect to Beta Cluster database deployment-db1 or deployment-db2 (MariaDB down) - https://phabricator.wikimedia.org/T96905#1242882 (10Ryasmeen) The issue has evolved a bit.Not getting those cant connect to MySQL error, but VE is silently failing to load there.There is one error in the cons... [19:13:25] 10Browser-Tests, 3Gather Sprint Forward, 6Mobile-Web, 10Mobile-Web-Sprint-45-Snakes-On-A-Plane, 5Patch-For-Review: Fix failed MobileFrontend browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94156#1242926 (10Jdlrobson) #627 passed. We'll open more specific bugs from now on e.g. T97460 [19:22:00] 10Beta-Cluster: Can't connect to Beta Cluster database deployment-db1 or deployment-db2 (MariaDB down) - https://phabricator.wikimedia.org/T96905#1242939 (10Krenair) That error looks like it could be related to https://gerrit.wikimedia.org/r/#/c/206033/ [19:22:34] 10Beta-Cluster: Can't connect to Beta Cluster database deployment-db1 or deployment-db2 (MariaDB down) - https://phabricator.wikimedia.org/T96905#1242943 (10Krenair) (So yes, I see the error as well, but VE still functions) [19:25:11] 10Beta-Cluster: Can't connect to Beta Cluster database deployment-db1 or deployment-db2 (MariaDB down) - https://phabricator.wikimedia.org/T96905#1242947 (10Ryasmeen) Krenair, try saving your edit and re-open multiple times . It didnt occur to me in one shot as well. [19:36:40] zeljkof: yes [19:39:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-Vagrant, 7Documentation: Document RSpec workflow on MediaWiki-Vagrant - https://phabricator.wikimedia.org/T97464#1243014 (10Tgr) 3NEW [19:54:17] YuviPanda: sorry, I have forgot my question :) what are you answering to? [19:55:02] zeljkof: IRCCLoud [19:55:28] YuviPanda: thanks, I am already in :) [20:10:09] 6Release-Engineering, 10MediaWiki-Vagrant, 7Documentation: Document RSpec workflow on MediaWiki-Vagrant - https://phabricator.wikimedia.org/T97464#1243014 (10hashar) [20:12:21] 6Release-Engineering, 10MediaWiki-Vagrant, 7Documentation: Document RSpec workflow on MediaWiki-Vagrant - https://phabricator.wikimedia.org/T97464#1243139 (10hashar) @dduvall would you mind adding a testing section in MWV README.md file? A step by step tutorial for dummies about how to run rspec tests. I g... [20:12:40] zeljkof: ah cool :) [20:23:58] 6Release-Engineering, 10MediaWiki-Vagrant, 7Documentation: Document RSpec workflow on MediaWiki-Vagrant - https://phabricator.wikimedia.org/T97464#1243177 (10Tgr) Didn't work for me: ``` vagrant@mediawiki-vagrant:/vagrant$ bundle install Fetching gem metadata from https://rubygems.org/.......... Fetching ver... [20:45:02] 5Continuous-Integration-Isolation: Instances created by Nodepool cant run puppet due to missing certificate - https://phabricator.wikimedia.org/T96670#1243249 (10hashar) For now, only the CI isolation project is going to spawn instances directly via the OpenStack API. In firstboot.sh, the only line relying on p... [20:46:36] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Isolation, 6operations, 7Nodepool: Create a Debian package for NodePool on Debian Jessie - https://phabricator.wikimedia.org/T89142#1243251 (10hashar) 5Open>3Resolved a:3hashar The preliminary work is completed. The package will be enha... [20:46:39] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Isolation: Puppetize Nodepool configuration - https://phabricator.wikimedia.org/T89143#1243254 (10hashar) [20:53:37] 10Continuous-Integration-Infrastructure, 5Continuous-Integration-Isolation, 6operations, 7Nodepool: Create a Debian package for NodePool on Debian Jessie - https://phabricator.wikimedia.org/T89142#1243271 (10hashar) I have poked the [[ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=781027 | Debian ITP ]... [20:54:01] a heads up to folks that tomorrow I plan to update salt in deployment-prep to 2014.7.5 (from 2014.7.1); starting at 12 noon UTC. This could take as long as two hours of fooling around, during which time salt and trebuchet may not be functional [20:54:25] tomorrow = Wednesday Apr 29 [20:55:34] YuviPanda: maybe you were asking if the lucid instance over there could be tossed. My answer is yes, I'm giving up on it [20:55:49] YES THAT WAS ME LET ME KILL IT WHEEE [20:55:57] we really just neeed to get the last prod host off of lucid [20:56:17] if it has salt issues so much the better, more impetus :-P [20:57:02] I will give notice in this channel tomorrow a little before I start [20:57:08] apergos: my labs lucid instance died and andrew couldn't revive it or let me make a new one :( [20:57:24] it was really ironic as it was a mailman replica which is the only lucid host in prod [20:57:30] yes it is [20:57:35] and we need to get off of it [20:57:45] really asap [20:57:48] hey if you volunteer ;) [20:57:58] the EOL for lucid is Thursday too [20:58:05] I know, that's why we need to get off [20:58:11] apergos: killed [20:58:14] thanks Yuvi [20:59:16] apergos: only way it'll go anywhere is a heroic op do it or hoping Faidon has spare time to do this :/ [21:00:08] 5Continuous-Integration-Isolation: Create a Trusty labs image for CI isolation project - https://phabricator.wikimedia.org/T97472#1243303 (10hashar) 3NEW [21:00:37] 5Continuous-Integration-Isolation: Create a Trusty labs image for CI isolation project - https://phabricator.wikimedia.org/T97472#1243310 (10hashar) + @Andrew since he is the grand master of image building for wmflabs ! [21:00:48] PROBLEM - Host deployment-lucid-salt is DOWN: CRITICAL - Host Unreachable (10.68.17.49) [21:00:53] 5Continuous-Integration-Isolation: Create a Trusty labs image for CI isolation project - https://phabricator.wikimedia.org/T97472#1243313 (10hashar) a:3hashar [21:01:17] JohnFLewis: well can't take it on right now so not going to even pretend to lick that cookie [21:01:35] oh noes deployment-lucid-salt is down! and it's critical! :-D :-D [21:01:50] c'mon shinken, get it together [21:02:09] apergos: which is probably what all ops will say :) I've tried to prod people to take it on multiple times yet everyone says the same 'not me. ask someone else' :/ [21:03:09] shinken-wm: I know, right? Isn’ it amazing that that host is down and never coming back up? :) [21:05:26] Project beta-scap-eqiad build #50835: FAILURE in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50835/ [21:05:33] 5Continuous-Integration-Isolation: Create a Trusty labs image for CI isolation project - https://phabricator.wikimedia.org/T97472#1243326 (10hashar) Created instance [[ https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000bad.eqiad.wmflabs | i-00000bad ]] with image "ubuntu-14.04-trusty" and hostname integra... [21:05:46] well, midnight, so time to sleep [21:05:51] apergos: night [21:05:59] have a good one y'all [21:12:56] apergos: night! [21:13:57] 21:04:48 Job ['/srv/deployment/scap/scap/bin/sync-common', '--no-update-l10n'] called with an empty host list. [21:14:12] greg-g: I think there is a task for that one [21:14:38] something like hosts that used to be shipped via a dsh file which is no more existent [21:14:58] Yippee, build fixed! [21:14:59] Project beta-scap-eqiad build #50836: FIXED in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50836/ [21:15:03] well, ok [21:15:21] the warning always appears [21:15:32] scap merely complains because it is not rsyncing to rsync proxies [21:15:37] but I dont think we use any on beta [21:15:45] instead scap rsync directly to the hosts [21:17:24] PROBLEM - Puppet staleness on deployment-eventlogging02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [21:19:15] indeed. that last beta-scap-eqiad failure was the hostkey changing on videoscaler01 [21:21:37] * greg-g nods [21:32:22] there was a proxy on beta until it was killed last quarter [21:32:27] rsync proxy that is [21:34:31] <^d> That should be easier to configure if my dsh patches to puppet get merged [21:34:33] <^d> yay hiera [21:34:46] <^d> (shameless ping to YuviPanda ^) [21:35:04] it was -2’d by _joe_ who hasn’t had time to look at it since :| [21:36:00] <^d> I could write a new patch and not put _joe_ on it ;-) [21:36:02] <^d> Sneaky! [21:36:59] <^d> Also, https://gerrit.wikimedia.org/r/#/c/206132/ is completely standalone [21:38:37] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:38:58] ^d: done [21:39:15] <^d> :D [21:50:38] sneaky [21:50:57] have sweet dreams everyone! [21:51:09] hashar: see ya! o/ [21:53:56] PROBLEM - Puppet failure on integration-vmbuilder-trusty is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:54:23] Project beta-scap-eqiad build #50841: FAILURE in 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50841/ [22:00:13] (03PS1) 10Dduvall: Field mappings for more build information [integration/raita] - 10https://gerrit.wikimedia.org/r/207291 [22:03:56] RECOVERY - Puppet failure on integration-vmbuilder-trusty is OK: OK: Less than 1.00% above the threshold [0.0] [22:04:54] Yippee, build fixed! [22:04:55] Project beta-scap-eqiad build #50842: FIXED in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/50842/ [22:05:31] fantastic determinism there. [23:00:34] Is it intentional that http://test.wikipedia.beta.wmflabs.org/ redirects to http://test.wikimedia.beta.wmflabs.org/wiki/Main_Page ? [23:00:42] Note second is wikiMedia. [23:17:59] !log Ran mysql> INSERT INTO sites (SELECT * FROM wikidatawiki.sites); on testwiki to populate the sites table [23:18:05] Logged the message, Master [23:18:22] matt_flaschen, that redirection itself sounds good [23:18:32] We really need to do that in production to be honest [23:18:49] !log Ran mysql> INSERT INTO sites (SELECT * FROM wikidatawiki.sites); on enwikinews to populate the sites table [23:18:53] Logged the message, Master [23:20:50] Krenair, why? [23:21:00] Just because it's not really a Wikipedia? [23:21:03] yes [23:21:15] I guess. Beta should be consistent with production, though. [23:21:31] yeah, I don't know if it's necessarily intentional here [23:30:42] Are any sites in beta https only? [23:31:38] nevermind, the script got that right [23:33:01] !log Ran foreachwiki extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --load-from 'http://meta.wikimedia.beta.wmflabs.org/w/api.php' to fix all sites tables [23:33:06] Logged the message, Master [23:34:31] hoo, should it work now, or does the memcached step still need to be done? [23:34:50] hoo, I thought beta was all http only? [23:34:59] matt_flaschen: I'm not sure that script purges memcached... it should, but it might not [23:35:00] try it [23:35:12] Krenair: Nope, apparently ruwiki in beta is https only [23:35:15] Wikinews still fails. [23:35:19] Haven't tried the script yet. [23:35:20] argh [23:35:42] Krenair: You're right [23:35:48] the sitematrix is wrong, doh [23:37:19] !log Ran foreachwiki extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --load-from 'http://meta.wikimedia.beta.wmflabs.org/w/api.php' --force-protocol http (because some sites are http only, although the sitematrix claims otherwise) [23:37:23] Logged the message, Master [23:38:25] matt_flaschen: How about now? [23:38:49] Yay! [23:39:11] Let me try the script now. [23:39:19] Wikinews works. [23:39:25] puh, nice [23:39:36] If the sitematrix is correct all wikis should be fine now [23:39:51] unless I missed more weird things in the config [23:41:39] Script works. [23:41:48] :)