[00:01:12] RECOVERY - Free space - all mounts on deployment-cache-upload04 is OK: OK: All targets OK [00:21:23] MaxSem, wanna update the ticket too? [00:21:38] haven't seen it [00:22:42] 10Beta-Cluster-Infrastructure: /srv/vdb almost full on deployment-cache-upload04 - https://phabricator.wikimedia.org/T146952#2676094 (10MaxSem) 05Open>03Resolved a:03MaxSem Nuked cache, rebooted just in case - now back to 59%. [00:50:34] PROBLEM - Puppet staleness on deployment-prometheus01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [00:57:43] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 07Nodepool: Investigate why Nodepool instances are sometime slow to reach READY state - https://phabricator.wikimedia.org/T146813#2676130 (10Paladox) This https://github.com/openstack-infra/nodepool/commit/e1bfe6dcb8b8540ecd3e513d725... [01:35:06] Project beta-scap-eqiad build #122062: 04FAILURE in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122062/ [01:45:01] Project beta-scap-eqiad build #122063: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122063/ [01:55:08] Project beta-scap-eqiad build #122064: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122064/ [02:05:03] Project beta-scap-eqiad build #122065: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122065/ [02:11:08] jdlrobson: wmf-config/mobile-labs.php still references wmgMFUseCentralAuthToken [02:11:29] https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122065/console [02:15:06] Project beta-scap-eqiad build #122066: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122066/ [02:20:00] Project beta-update-databases-eqiad build #11689: 04FAILURE in 0.49 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11689/ [02:25:05] Project beta-scap-eqiad build #122067: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122067/ [02:35:02] Project beta-scap-eqiad build #122068: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122068/ [02:45:09] Project beta-scap-eqiad build #122069: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122069/ [02:48:59] 06Release-Engineering-Team, 10TimedMediaHandler-Player, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: ogv.js player fails due to extension files being from newer version of MediaWiki than is deployed - https://phabricator.wikimedia.org/T146363#2676161 (10Krinkle) [02:55:08] Project beta-scap-eqiad build #122070: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122070/ [03:05:03] Project beta-scap-eqiad build #122071: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122071/ [03:15:03] Project beta-scap-eqiad build #122072: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122072/ [03:20:00] Project beta-update-databases-eqiad build #11690: 04STILL FAILING in 0.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11690/ [03:25:01] Project beta-scap-eqiad build #122073: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122073/ [03:35:00] Project beta-scap-eqiad build #122074: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122074/ [03:45:01] Project beta-scap-eqiad build #122075: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122075/ [03:54:59] Project beta-scap-eqiad build #122076: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122076/ [04:05:02] Project beta-scap-eqiad build #122077: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122077/ [04:14:59] Project beta-scap-eqiad build #122078: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122078/ [04:15:23] PROBLEM - Puppet staleness on deployment-db1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [04:20:00] Project beta-update-databases-eqiad build #11691: 04STILL FAILING in 0.55 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11691/ [04:25:01] Project beta-scap-eqiad build #122079: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122079/ [04:27:02] PROBLEM - Puppet staleness on deployment-db2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [04:29:44] Yippee, build fixed! [04:29:44] Project selenium-Core ยป firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #169: 09FIXED in 6 min 12 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/169/ [04:35:03] Project beta-scap-eqiad build #122080: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122080/ [04:45:02] Project beta-scap-eqiad build #122081: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122081/ [04:55:03] Project beta-scap-eqiad build #122082: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122082/ [05:05:00] Project beta-scap-eqiad build #122083: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122083/ [05:15:02] Project beta-scap-eqiad build #122084: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122084/ [05:20:00] Project beta-update-databases-eqiad build #11692: 04STILL FAILING in 0.57 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11692/ [05:25:01] Project beta-scap-eqiad build #122085: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122085/ [05:34:59] Project beta-scap-eqiad build #122086: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122086/ [05:45:00] Project beta-scap-eqiad build #122087: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122087/ [05:55:00] Project beta-scap-eqiad build #122088: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122088/ [06:05:05] Project beta-scap-eqiad build #122089: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122089/ [06:14:42] Project beta-code-update-eqiad build #123521: 04FAILURE in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/123521/ [06:16:41] * legoktm clicks [06:17:01] 05:05:00 [4159463b1d3f0ec4c435871c] [no req] InvalidArgumentException from line 133 of /srv/mediawiki-staging/php-master/includes/objectcache/ObjectCache.php: Invalid object cache type "Array" requested. It is not present in $wgObjectCaches. [06:20:04] Project beta-update-databases-eqiad build #11693: 04STILL FAILING in 4.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11693/ [06:24:42] Project beta-code-update-eqiad build #123522: 04STILL FAILING in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/123522/ [06:34:45] Yippee, build fixed! [06:34:45] Project beta-code-update-eqiad build #123523: 09FIXED in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/123523/ [06:35:06] Project beta-scap-eqiad build #122090: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122090/ [06:44:59] Project beta-scap-eqiad build #122091: 04STILL FAILING in 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122091/ [06:55:02] Project beta-scap-eqiad build #122092: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122092/ [07:05:00] Project beta-scap-eqiad build #122093: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122093/ [07:15:01] Project beta-scap-eqiad build #122094: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122094/ [07:20:00] Project beta-update-databases-eqiad build #11694: 04STILL FAILING in 0.45 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11694/ [07:25:08] Project beta-scap-eqiad build #122095: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122095/ [07:35:00] Project beta-scap-eqiad build #122096: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122096/ [07:45:08] Project beta-scap-eqiad build #122097: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122097/ [07:55:02] Project beta-scap-eqiad build #122098: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122098/ [08:05:02] Project beta-scap-eqiad build #122099: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122099/ [08:09:56] gonna look at ^^ [08:12:33] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:15:07] Project beta-scap-eqiad build #122100: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122100/ [08:20:01] Project beta-update-databases-eqiad build #11695: 04STILL FAILING in 0.56 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11695/ [08:25:00] Project beta-scap-eqiad build #122101: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122101/ [08:31:16] I can reproduce ! :] [08:32:44] hashar: Want me to deploy my change from last night? :P [08:35:01] Project beta-scap-eqiad build #122102: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122102/ [08:35:40] (03PS1) 10Legoktm: doc: Add ScopedCallback [integration/docroot] - 10https://gerrit.wikimedia.org/r/313371 (https://phabricator.wikimedia.org/T146258) [08:36:51] (03CR) 10Legoktm: [C: 032] doc: Add ScopedCallback [integration/docroot] - 10https://gerrit.wikimedia.org/r/313371 (https://phabricator.wikimedia.org/T146258) (owner: 10Legoktm) [08:37:17] (03Merged) 10jenkins-bot: doc: Add ScopedCallback [integration/docroot] - 10https://gerrit.wikimedia.org/r/313371 (https://phabricator.wikimedia.org/T146258) (owner: 10Legoktm) [08:37:43] Project beta-scap-eqiad build #122103: 04STILL FAILING in 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122103/ [08:38:24] (03PS1) 10Legoktm: Whitelist Fomafix [integration/config] - 10https://gerrit.wikimedia.org/r/313372 [08:39:49] 08:37:43 Notice: Undefined index: wmgExtraLanguageNames in /srv/mediawiki-staging/php-master/includes/SiteConfiguration.php on line 312 [08:40:35] 10Continuous-Integration-Infrastructure: Create pretty landing page at https://doc.wikimedia.org/cover/ - https://phabricator.wikimedia.org/T146970#2676464 (10Legoktm) [08:45:01] Project beta-scap-eqiad build #122104: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122104/ [08:47:30] Reedy: yeah moaaar work has to be polished up :D [08:47:37] I found the issue with beta-scap [08:47:44] computers suck [08:47:47] some rewrite of the way we initiazie object cache [08:47:47] that's always the issue [08:55:01] Project beta-scap-eqiad build #122105: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122105/ [08:59:52] !log Hopefully going to get beta fixed via mw/core revert patch https://gerrit.wikimedia.org/r/313373 [08:59:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:02:47] Yippee, build fixed! [09:02:47] Project beta-scap-eqiad build #122106: 09FIXED in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122106/ [09:03:14] heh [09:10:44] \o/ [09:11:34] (03CR) 10Hashar: [C: 032] Whitelist Fomafix [integration/config] - 10https://gerrit.wikimedia.org/r/313372 (owner: 10Legoktm) [09:12:32] (03Merged) 10jenkins-bot: Whitelist Fomafix [integration/config] - 10https://gerrit.wikimedia.org/r/313372 (owner: 10Legoktm) [09:15:04] (03PS5) 10Hashar: Ignore refs/meta/config refs from jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/311032 (https://phabricator.wikimedia.org/T52389) (owner: 10Paladox) [09:15:36] (03CR) 10Hashar: [C: 032] "Thank you !!!!!! :)" [integration/config] - 10https://gerrit.wikimedia.org/r/311032 (https://phabricator.wikimedia.org/T52389) (owner: 10Paladox) [09:16:13] (03Merged) 10jenkins-bot: Ignore refs/meta/config refs from jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/311032 (https://phabricator.wikimedia.org/T52389) (owner: 10Paladox) [09:18:45] (03PS1) 10Hashar: Revert "Ignore refs/meta/config refs from jenkins" [integration/config] - 10https://gerrit.wikimedia.org/r/313377 [09:18:53] (03CR) 10Hashar: [C: 032] Revert "Ignore refs/meta/config refs from jenkins" [integration/config] - 10https://gerrit.wikimedia.org/r/313377 (owner: 10Hashar) [09:19:32] (03CR) 10Hashar: "I have been too fast. That would cause Zuul to trigger jobs for any ref-update such as a tip of branch being updated after a change is mer" [integration/config] - 10https://gerrit.wikimedia.org/r/311032 (https://phabricator.wikimedia.org/T52389) (owner: 10Paladox) [09:20:37] (03Merged) 10jenkins-bot: Revert "Ignore refs/meta/config refs from jenkins" [integration/config] - 10https://gerrit.wikimedia.org/r/313377 (owner: 10Hashar) [09:20:55] Yippee, build fixed! [09:20:56] Project beta-update-databases-eqiad build #11696: 09FIXED in 55 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/11696/ [09:22:34] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [09:42:51] 06Release-Engineering-Team, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2676630 (10jcrespo) @mmodell Thank you very much for investigating, if today is "phabricator's mainten... [09:53:11] 06Release-Engineering-Team, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2676675 (10Paladox) @jcrespo and @mmodell I found that innodb full search is more accurate then myisam... [09:54:24] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 15User-zeljkofilipin: mwext-mw-selenium jenkins job not aborting after 30 minutes - https://phabricator.wikimedia.org/T146903#2676679 (10hashar) https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1013/ w... [09:57:26] (03CR) 10Paladox: "Better solution is I025a0fa3ebc29b371763b4bb7894b89193118f9d which I tested, I also wrote back to Krinkle there saying I tested it and it " [integration/config] - 10https://gerrit.wikimedia.org/r/311032 (https://phabricator.wikimedia.org/T52389) (owner: 10Paladox) [09:57:41] (03CR) 10Paladox: "@hashar ^*" [integration/config] - 10https://gerrit.wikimedia.org/r/311032 (https://phabricator.wikimedia.org/T52389) (owner: 10Paladox) [10:01:55] Project beta-scap-eqiad build #122112: 15ABORTED in 7 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122112/ [10:05:18] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 15User-zeljkofilipin: mwext-mw-selenium jenkins job not aborting after 30 minutes - https://phabricator.wikimedia.org/T146903#2676704 (10hashar) From https://integration.wikimedia.org/ci/monitoring?part=threads the thread o... [10:06:27] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 15User-zeljkofilipin: mwext-mw-selenium jenkins job not aborting after 30 minutes - https://phabricator.wikimedia.org/T146903#2676707 (10Paladox) Oh, could that be because we use the old ssh plugin? Should we report this as... [10:09:52] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 15User-zeljkofilipin: mwext-mw-selenium jenkins job not aborting after 30 minutes - https://phabricator.wikimedia.org/T146903#2676712 (10hashar) The instance integration-slave-trusty-1013.integration.eqiad.wmflabs is unrea... [10:10:11] PROBLEM - Host integration-slave-trusty-1013 is DOWN: CRITICAL - Host Unreachable (10.68.18.28) [10:11:01] RECOVERY - Host integration-slave-trusty-1013 is UP: PING OK - Packet loss = 0%, RTA = 0.86 ms [10:11:30] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 15User-zeljkofilipin: mwext-mw-selenium jenkins job not aborting after 30 minutes - https://phabricator.wikimedia.org/T146903#2676723 (10Paladox) I think it is because jenkins dosent check to make sure that the instance has... [10:15:05] PROBLEM - Puppet staleness on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [43200.0] [10:15:35] Project beta-scap-eqiad build #122114: 15ABORTED in 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/122114/ [10:25:08] RECOVERY - Puppet staleness on integration-slave-trusty-1013 is OK: OK: Less than 1.00% above the threshold [3600.0] [10:28:44] !log Upgrading Jenkins plugins with zeljkof :] [10:28:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:37:51] !log Jenkins upgrade AnsiColor plugin from 0.3.1 to 0.4.2 [10:37:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:38:02] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 15User-zeljkofilipin: mwext-mw-selenium jenkins job not aborting after 30 minutes - https://phabricator.wikimedia.org/T146903#2676779 (10zeljkofilipin) 05Open>03Resolved a:03zeljkofilipin Everything looks fine now. Lo... [10:48:22] (03PS1) 10Legoktm: Add jobs for mediawiki/libs/Timestamp [integration/config] - 10https://gerrit.wikimedia.org/r/313384 [10:48:52] (03CR) 10Legoktm: [C: 032] Add jobs for mediawiki/libs/Timestamp [integration/config] - 10https://gerrit.wikimedia.org/r/313384 (owner: 10Legoktm) [10:49:52] (03Merged) 10jenkins-bot: Add jobs for mediawiki/libs/Timestamp [integration/config] - 10https://gerrit.wikimedia.org/r/313384 (owner: 10Legoktm) [10:50:57] !log deploying https://gerrit.wikimedia.org/r/313384 [10:51:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:53:30] 10Gerrit, 07Zuul: Gerrit ssh command failing on new repositories, causing zuul to not run against a change - https://phabricator.wikimedia.org/T146260#2676793 (10Legoktm) I'm fairly sure it only starts working on the third commit to a repo. The second commit for Timestamp's command is `gerrit query --format j... [10:53:41] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Zuul: Gerrit ssh command failing on new repositories, causing zuul to not run against a change - https://phabricator.wikimedia.org/T146260#2676794 (10Legoktm) [11:30:14] (03PS1) 10Hashar: Filter out refs/meta/config from all pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/313387 (https://phabricator.wikimedia.org/T52389) [11:42:38] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Upstream, 07Zuul: Gerrit ssh command failing on new repositories, causing zuul to not run against a change - https://phabricator.wikimedia.org/T146260#2676887 (10hashar) That is #upstream bug https://bugs.chromium.org/p/gerrit/issues/detail?id=4006 relat... [11:42:47] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Upstream, 07Zuul: Gerrit ssh command failing on new repositories, causing zuul to not run against a change - https://phabricator.wikimedia.org/T146260#2676889 (10hashar) p:05Triage>03Normal [11:45:19] 10Gerrit: git-review can show of hundreds of commits when someone tries to make a simple commit and it's terrifying - https://phabricator.wikimedia.org/T59399#2676897 (10hashar) 05Open>03declined I am declining this 3 years old issue. The fix is either: * have all remotes updated before using the command *... [11:50:03] (03CR) 10Paladox: [C: 031] "We should definitely try this. Haven't tried it though but it looks all ok. :)" [integration/config] - 10https://gerrit.wikimedia.org/r/313387 (https://phabricator.wikimedia.org/T52389) (owner: 10Hashar) [11:51:31] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 15User-zeljkofilipin: mwext-mw-selenium jenkins job not aborting after 30 minutes - https://phabricator.wikimedia.org/T146903#2676903 (10Paladox) Oh, but coulden jenkins get extra functionality to detect instances that have... [11:53:42] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 15User-zeljkofilipin: mwext-mw-selenium jenkins job not aborting after 30 minutes - https://phabricator.wikimedia.org/T146903#2676904 (10hashar) The instance died while the job was running ... [11:54:27] 10Beta-Cluster-Infrastructure: Beta cluster database update job is stall, because of Swift? - https://phabricator.wikimedia.org/T146947#2676905 (10hashar) Thanks Alex! [11:54:50] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 07Jenkins, 15User-zeljkofilipin: mwext-mw-selenium jenkins job not aborting after 30 minutes - https://phabricator.wikimedia.org/T146903#2676906 (10Paladox) Oh [12:04:21] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Upstream, 07Zuul: Gerrit ssh command failing on new repositories, causing zuul to not run against a change - https://phabricator.wikimedia.org/T146260#2655194 (10Paladox) Fixed in gerrit 2.12.5. [12:05:25] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Upstream, 07Zuul: Gerrit ssh command failing on new repositories, causing zuul to not run against a change - https://phabricator.wikimedia.org/T146260#2676913 (10Paladox) [12:05:27] 10Gerrit: Update gerrit to 2.12.5 - https://phabricator.wikimedia.org/T143089#2676912 (10Paladox) [12:11:15] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Upstream, 07Zuul: Gerrit ssh command failing on new repositories, causing zuul to not run against a change - https://phabricator.wikimedia.org/T146260#2676919 (10Paladox) Can we remove the Zuul and continuous projects please? Since this is a gerrit error... [12:38:47] PROBLEM - Keyholder status on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:51:46] !log Restarted udp2log on deployment-fluorine02 [13:51:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:52:06] !log Restarted jobrunner / jobchron on deployment-jobrunner02 . Were no more logging to /var/log/mediawiki/ somehow [13:52:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:55:48] PROBLEM - Puppet run on deployment-fluorine02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [14:00:47] RECOVERY - Puppet run on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:37] PROBLEM - Puppet run on deployment-jobrunner02 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [14:15:36] RECOVERY - Puppet run on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:43:09] !log logstash-beta: refreshed the field list via https://logstash-beta.wmflabs.org/app/kibana#/settings/indices/logstash [15:43:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:31:29] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T145220#2677383 (10greg) a:03thcipriani [16:32:08] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T146998#2677385 (10greg) [16:32:27] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T145220#2623702 (10greg) [17:05:59] 10Continuous-Integration-Config, 10RESTBase, 06Wikipedia-Android-App-Backlog: Kick off periodic Android CI tests when RESTBase is updated on beta labs - https://phabricator.wikimedia.org/T146488#2677546 (10Niedzielski) Sorry for the late response on this. Yes, I broke CI recently and am working to fix it. If... [17:09:42] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth, 10MobileFrontend, 06Reading-Web-Backlog, 13Patch-For-Review: beta cluster: Notice: Undefined variable: wmgMFUseCentralAuthToken in /srv/mediawiki-staging/wmf-config/mobile-labs.php on line... - https://phabricator.wikimedia.org/T146945#2677564 [17:19:32] PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:45:41] ebernhardson hi, is there any improvements between elasticsearch 1 and 2? [17:45:55] Since im testing this on phab using elasticsearch 1 [17:45:57] 1.7.2 [17:45:58] paladox: did you look up its ChangeLog? [17:46:02] Yes [17:46:15] and that did not answer your question? [17:46:24] Well that had manly api changes [17:46:40] It didn't tell you correctly what new features there are [17:47:02] so "features" are "improvements" to you? [17:47:15] api changes can also easily be improvements. [17:47:24] Yes [17:47:50] so, question answered? [17:48:36] paladox: there are lots of new features in 2.x, but they arnt "free" features, they are things that can be used [17:48:47] Oh [17:48:52] thanks [17:50:02] when it comes to elasticsearch vs mysql for search, my impression (caveat, i don't know how mysql text search works) is that elasticsearch has much more power than mysql, but they are things that have to be explicitly chosen to be used. To take an example the search queries we send to elasticsearch from mediawiki are multiple pages of json (pretty printed) [17:50:17] if phabricator only has a naive usage of elasticsearch, it's probably only marginally better than mysql, if at all [17:50:58] Oh [17:51:19] We tryed elasticsearch before, but we are retrying it with https://phabricator.wikimedia.org/T146843 [18:24:21] PROBLEM - Host integration-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.16.42) [19:41:34] 03Scap3 (Scap3-MediaWiki-MVP), 10scap, 07Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597#1883295 (10greg) This follow-up task from an incident report has not been updated recently. If it is no longer valid, please add a comment explaining why. If it is s... [19:42:05] 10Continuous-Integration-Config, 06Operations, 07Regression, 07Wikimedia-Incident: operations-apache-config-lint replacement doesn't check syntax - https://phabricator.wikimedia.org/T114801#2678045 (10greg) This follow-up task from an incident report has not been updated recently. If it is no longer valid,... [19:42:28] 10Continuous-Integration-Config, 07Wikimedia-Incident: add a check for whitespace before leading 10Beta-Cluster-Infrastructure, 10Staging, 07Wikimedia-Incident: Rework beta apache config - https://phabricator.wikimedia.org/T1256#2678073 (10greg) This follow-up task from an incident report has not been updated recently. If it is no longer valid, please add a comment explaining why. If it is still valid,... [19:49:48] 06Release-Engineering-Team, 15User-greg: Ping tasks in #wikimedia-incident without recent activity for follow-up near the end of FY1617Q1 - https://phabricator.wikimedia.org/T144973#2678103 (10greg) 05Open>03Resolved Done. Hit these: https://phabricator.wikimedia.org/maniphest/?ids=138562,137625,137350,13... [19:50:37] 06Release-Engineering-Team, 15User-greg: Ping tasks in #wikimedia-incident without recent activity for follow-up near the end of FY1617Q2 - https://phabricator.wikimedia.org/T147022#2678107 (10greg) [19:50:45] ebernhardson: indeed phabricator doesn't have a very advanced elasticsearch integration, but it leaves room for improvement and mysql search is kind of a dead-end [19:51:01] 06Release-Engineering-Team, 15User-greg: Ping tasks in #wikimedia-incident without recent activity for follow-up near the end of FY1617Q2 - https://phabricator.wikimedia.org/T147022#2678123 (10greg) [20:33:38] 06Release-Engineering-Team, 10Phabricator, 13Patch-For-Review, 07Wikimedia-Incident: Contention on search phabricator database creating full phabricator outages - https://phabricator.wikimedia.org/T146673#2678299 (10mmodell) @jcrespo The maintenance window was last night. I would be happy to work with you... [20:33:56] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review: Fix MobileFrontend scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily - https://phabricator.wikimedia.org/T94156#2678303 (10jhobs) [20:33:58] 10Browser-Tests-Infrastructure, 10MobileFrontend, 06Reading-Web-Backlog, 07Browser-Tests, 15User-zeljkofilipin: `Generic special page features.Search from Watchlist` test failing with Net::ReadTimeout - https://phabricator.wikimedia.org/T130971#2678301 (10jhobs) 05Open>03Invalid No longer occurring. [20:46:32] 06Release-Engineering-Team, 15User-greg: skill matrix updates - https://phabricator.wikimedia.org/T140507#2678348 (10greg) [20:46:34] 06Release-Engineering-Team, 15User-greg: Update RelEng skillmatrix (due 10/13) - https://phabricator.wikimedia.org/T146516#2678347 (10greg) [21:09:26] 10Gerrit: Gerrit account issue (can't log into Gerrit, can log into wikitech) - https://phabricator.wikimedia.org/T146887#2678374 (10demon) 05Open>03Resolved a:03demon Fixed. Not entirely sure I remember why the account was deactivated. [21:22:36] PROBLEM - Keyholder status on deployment-mira is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:47:37] RECOVERY - Keyholder status on deployment-mira is OK: OK: Less than 100.00% above the threshold [0.0] [21:58:44] RECOVERY - Keyholder status on deployment-tin is OK: OK: Less than 100.00% above the threshold [0.0] [22:20:15] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Upstream: Gerrit ssh command failing on new repositories, causing zuul to not run against a change - https://phabricator.wikimedia.org/T146260#2678542 (10Paladox) [22:20:24] 10Continuous-Integration-Infrastructure, 10Gerrit, 07Upstream: Gerrit ssh command failing on new repositories, causing zuul to not run against a change - https://phabricator.wikimedia.org/T146260#2655194 (10Paladox) Has nothing to do with zuul