[00:17:02] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #228: 04FAILURE in 1 min 2 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/228/ [00:20:05] Project beta-update-databases-eqiad build #13284: 04STILL FAILING in 5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13284/ [01:20:05] Project beta-update-databases-eqiad build #13285: 04STILL FAILING in 4.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13285/ [02:17:28] Yippee, build fixed! [02:17:29] Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #239: 09FIXED in 4 min 28 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/239/ [02:20:06] Project beta-update-databases-eqiad build #13286: 04STILL FAILING in 6.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13286/ [03:20:05] Project beta-update-databases-eqiad build #13287: 04STILL FAILING in 4.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13287/ [04:19:03] Yippee, build fixed! [04:19:04] Project selenium-MultimediaViewer » chrome,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #224: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/224/ [04:20:06] Project beta-update-databases-eqiad build #13288: 04STILL FAILING in 5.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13288/ [05:18:26] 10Continuous-Integration-Infrastructure, 06Labs: Do contintcloud and other CI boxes know about labs-ns1? - https://phabricator.wikimedia.org/T152370#2846099 (10Andrew) [05:20:06] Project beta-update-databases-eqiad build #13289: 04STILL FAILING in 5.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13289/ [06:20:06] Project beta-update-databases-eqiad build #13290: 04STILL FAILING in 6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13290/ [07:20:05] Project beta-update-databases-eqiad build #13291: 04STILL FAILING in 5.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13291/ [08:20:05] Project beta-update-databases-eqiad build #13292: 04STILL FAILING in 4.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13292/ [08:28:52] PROBLEM - Host hashar-deleteme is DOWN: CRITICAL - Host Unreachable (10.68.17.40) [08:29:00] ^^^ I have deleted it [08:34:35] LOL [08:35:32] :D [08:35:52] !log Pushing new Jessie image to Nodepool that is supposedly boot 3x times faster T113342 [08:35:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:36:28] and updating the snapshot image... [08:44:30] !log Image ci-jessie-wikimedia-1480926961 in wmflabs-eqiad is ready T113342 [08:44:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:50:18] 10Continuous-Integration-Infrastructure, 06Labs: Do contintcloud and other CI boxes know about labs-ns1? - https://phabricator.wikimedia.org/T152370#2846271 (10hashar) I have looked at it / filled a task about it ages ago but can not find it anymore. The issue is the DHCP server on labs only yield a single DN... [09:00:15] 10Continuous-Integration-Infrastructure, 06Labs: Do contintcloud and other CI boxes know about labs-ns1? - https://phabricator.wikimedia.org/T152370#2846279 (10hashar) Found it. T137460#2383979 and others have all the details. Namely the DHCP lease has: ``` option domain-name-servers 208.80.155.118; ``` And i... [09:00:33] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 10MediaWiki-Unit-tests: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org" - https://phabricator.wikimedia.org/T137460#2368900 (10hashar) [09:00:35] 10Continuous-Integration-Infrastructure, 06Labs: Do contintcloud and other CI boxes know about labs-ns1? - https://phabricator.wikimedia.org/T152370#2846285 (10hashar) [09:02:58] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 10MediaWiki-Unit-tests: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedia.org") - https://phabricator.wikimedia.org/T137460#2846286 (10ha... [09:20:05] Project beta-update-databases-eqiad build #13293: 04STILL FAILING in 4.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13293/ [09:39:14] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846335 (10hashar) [09:39:33] !log beta-update-databases-eqiad fails due to CONTENT_MODEL_FLOW_BOARD not registered on the wiki. T152379 [09:39:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:41:21] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846352 (10hashar) Probably caused by Flow change 480c26d8955ce7b549... [09:43:13] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846363 (10hashar) A couple PHP notice were already emitted in the p... [09:52:01] !log add https://gerrit.wikimedia.org/r/#/c/324642/ to the deployment-prep's puppet master to test nutcracker [09:52:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:52:05] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846383 (10hashar) The CONTENT_MODEL_FLOW_BOARD notice appeared with... [09:52:05] hashar: --^ [09:56:18] elukey: hello! good luck with the side effects :} [09:56:37] I have no clue how stats are retrieved, but I guess it just from some locally running prometheus/diamond collector [09:56:51] beware that if ones target localhost and the target is on 127.0.0.1 [09:56:58] the client might well resolves "localhost" has ::1 [09:57:07] and fails to connect if the daemon is ipv4 only [09:57:14] happened to me on some python based soft [09:58:39] good point, thanks :) [09:58:47] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2846404 (10hashar) I have deployed the new image. We will see on https://grafana.wikimedia.org/dashbo... [09:59:33] morning hashar [10:00:58] legoktm: yea :} [10:01:06] I have see your poolcoutner / rake / cucumber changes [10:01:11] gotta think about it [10:01:25] maybe the easiest is to run debian-glue [10:01:28] ah! okay, I don't even have to ping you about it then :P thank you [10:01:31] and have it run the tests in the context of the distro [10:01:47] not sure what is going to be the impact of adding rake/cucumber on all slaves [10:02:10] I thought about that but IIRC debian-glue needs the debian package to be in the root of the git repo? [10:02:19] this one is in the daemon/ directory [10:02:20] oh [10:02:37] guess that is hackable [10:03:23] not sure whether the deb package rules trigger the tests though [10:03:36] running dpkg-buildpackage does run `make test` [10:04:01] ideally the daemon should be in a separate repo since it's a service, but that was more work than I wanted to take on (I was just trying to get it to run on debian stretch! :P) [10:04:02] neat [10:04:23] if I get time this afternoon, I will hack a debian glue job that cd daemon [10:04:28] and maybe that will be enough [10:04:40] at worse I just copy paste the template hehe [10:05:11] :D [10:06:28] the other thing I was thinking about doing soon(TM) was creating a simple dashboard that listed people with 2+ merged patches in gerrit but weren't whitelisted yet [10:06:36] I think that would especially help right now during GCI [10:08:12] * legoktm zzz, night! [10:20:09] Project beta-update-databases-eqiad build #13294: 04STILL FAILING in 9.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13294/ [10:46:11] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846461 (10Krenair) Related commits: https://gerrit.wikimedia.org/r/#/... [11:09:13] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2846487 (10hashar) Should Aldo dosable thé puppet agent while at it. [11:20:06] Project beta-update-databases-eqiad build #13295: 04STILL FAILING in 5.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13295/ [11:20:33] 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 13Patch-For-Review: Speed up the time to get a Nodepool instances to achieve READY state - https://phabricator.wikimedia.org/T113342#2846513 (10hashar) I have booted a new instance and the `eth1` file is no more around: ``` $ find /et... [11:20:55] lunch & [11:27:13] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846525 (10Reedy) [11:31:29] Imma in ur wikis, fixing ur brokens [11:36:46] 11:31:20 ...Update 'FlowPopulateLinksTables' already logged as completed. [11:36:51] * Reedy watches the spinner [11:55:39] 10Continuous-Integration-Config, 06Discovery, 10Wikimedia-Portals: CI tests on wikimedia/portals repo: cache node_modules to save time - https://phabricator.wikimedia.org/T152386#2846564 (10MarcoAurelio) [11:58:10] 10Continuous-Integration-Config, 06Discovery, 10Wikimedia-Portals: wikimedia/portals repo might be using outdated and or deprecated tests in jenkins - https://phabricator.wikimedia.org/T152351#2846580 (10MarcoAurelio) p:05Triage>03High [11:58:28] 10Continuous-Integration-Config, 06Discovery, 10Wikimedia-Portals: wikimedia/portals repo might be using outdated and or deprecated tests in jenkins - https://phabricator.wikimedia.org/T152351#2845474 (10MarcoAurelio) @hashar: Fork done at T152386 [12:16:02] Project beta-update-databases-eqiad build #13296: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13296/ [12:49:10] 10scap: scap sync-l10n AttributeError: 'Namespace' object has no attribute 'message' - https://phabricator.wikimedia.org/T152390#2846662 (10Addshore) [13:05:00] Project beta-update-databases-eqiad build #13297: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13297/ [13:08:29] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846697 (10Reedy) 'FlowFixLog:version2' seems to be broken now, done t... [13:19:09] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846719 (10Paladox) It seems that we may need to revert, I found that... [13:19:34] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846721 (10Paladox) p:05Triage>03High [13:21:21] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846335 (10Paladox) https://phabricator.wikimedia.org/diffusion/EFLW/b... [13:30:23] (03PS1) 10Hashar: Fix up seb35 email [integration/config] - 10https://gerrit.wikimedia.org/r/325296 [13:32:27] (03CR) 10Hashar: [C: 032] Fix up seb35 email [integration/config] - 10https://gerrit.wikimedia.org/r/325296 (owner: 10Hashar) [13:33:18] (03Merged) 10jenkins-bot: Fix up seb35 email [integration/config] - 10https://gerrit.wikimedia.org/r/325296 (owner: 10Hashar) [13:46:54] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #234: 04FAILURE in 2 min 53 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/234/ [13:48:49] 05Gerrit-Migration, 10Differential: Find way to use Differential with plain git (i.e.: without requiring arc) - https://phabricator.wikimedia.org/T127#2846758 (10mmodell) Upstream has indicated that they will be working on https://secure.phabricator.com/T5000 soon. [14:05:00] Project beta-update-databases-eqiad build #13298: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13298/ [14:23:54] 03Scap3: scap sync-l10n AttributeError: 'Namespace' object has no attribute 'message' - https://phabricator.wikimedia.org/T152390#2846847 (10thcipriani) p:05Triage>03High [14:43:55] PROBLEM - Puppet run on repository is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:52:14] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2846965 (10Reedy) >>! In T152379#2846721, @Paladox wrote: > https://ph... [15:05:00] Project beta-update-databases-eqiad build #13299: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13299/ [15:14:57] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2847118 (10Paladox) Oh didn't notice that line, sorry. [15:23:55] RECOVERY - Puppet run on repository is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:49] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:51:03] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:05:00] Project beta-update-databases-eqiad build #13300: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13300/ [16:18:59] 10Beta-Cluster-Infrastructure: deployment-mx has old puppetmaster stuff - https://phabricator.wikimedia.org/T152353#2845596 (10hashar) The only reason I can think of is that deployment-mx has been made a standalone puppetmaster at some point, then the class got removed and that has been purged. The instance poin... [16:46:28] 10Continuous-Integration-Config, 10MediaWiki-extensions-PoolCounter, 13Patch-For-Review: PoolCounter daemon CI tests should use dependencies from debian, not bundler - https://phabricator.wikimedia.org/T152338#2845039 (10hashar) We are using http://jenkins-debian-glue.org/docs/ it has a script generate-git-s... [16:58:29] Project beta-update-databases-eqiad build #13301: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13301/ [17:00:15] (03PS1) 10Hashar: (WIP) (WIP) PoolCounter-debian-glue (WIP) (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/325322 (https://phabricator.wikimedia.org/T152338) [17:01:56] hashar: all jobs at zuul are queued with no progress, can you please have a look? Tnx. [17:02:21] sure [17:02:30] nodepool looks down, since i see no nodepool jobs running [17:02:55] looks like bunch of instances have been consumed https://grafana.wikimedia.org/dashboard/db/nodepool?panelId=1&fullscreen&from=now-3h&to=now [17:03:27] maybe wikimedia/portals patches that takes up to 12 minutes to build [17:03:31] :( [17:04:05] yeah that one is a problem [17:04:13] gotta find a good caching strategy for that repo [17:04:20] https://integration.wikimedia.org/ci/ shows no nodepool slave pooled [17:04:20] I've opened a task as you suggested [17:04:25] so they have all been consumed [17:04:43] and the grafana link above has a lot of yellow (instances are being build) and purple (nodepool deleting them) [17:04:58] I guess all instances got consumed quite fast [17:05:03] 10Browser-Tests-Infrastructure: Release new version of mediawiki_selenium - https://phabricator.wikimedia.org/T152422#2847579 (10zeljkofilipin) [17:05:26] 10Browser-Tests-Infrastructure, 15User-zeljkofilipin: Release new version of mediawiki_selenium - https://phabricator.wikimedia.org/T152422#2847594 (10zeljkofilipin) p:05Triage>03Normal [17:07:52] mafk: something happened somewhere [17:08:07] either we lost connection to the mysql database (which nodepool uses to track instances) [17:08:13] it is working now afaics [17:08:15] or labs api had some sort of time out that confusednodepool [17:08:18] seems it is back around [17:09:10] I think ^^ is fixed in a newer version since it dosent just use mysql to track, i think it added something else [17:09:14] but i forgot what it is [17:09:46] looks good now [17:09:53] I am heading back to my conf call [17:15:27] 10scap: ElectronPdfService mw extension l10n messages missing after full scap sync - https://phabricator.wikimedia.org/T152424#2847642 (10Addshore) [17:20:30] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 06Services (watching): Move primary trending service development to github - https://phabricator.wikimedia.org/T151469#2847684 (10hashar) This has been brought during the releng meeting. What needs to be do... [17:43:29] Project beta-update-databases-eqiad build #13302: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13302/ [17:53:37] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 06Services (watching): Move primary trending service development to github - https://phabricator.wikimedia.org/T151469#2847829 (10Pchelolo) >>! In T151469#2847684, @hashar wrote: >A bit harder would be to p... [17:57:10] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 06Services (watching): Move primary trending service development to github - https://phabricator.wikimedia.org/T151469#2847842 (10hashar) Neat! So if we have Kafka + Zookeeper packages installed in the CI i... [17:59:08] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 06Services (watching): Move primary trending service development to github - https://phabricator.wikimedia.org/T151469#2847846 (10Pchelolo) @hashar Actually, we already have a `packages.pp` class for trendi... [18:00:32] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 06Services (watching): Move primary trending service development to github - https://phabricator.wikimedia.org/T151469#2847863 (10Ottomata) Oh cool! CC @mforns @Nuria [18:02:03] 10Beta-Cluster-Infrastructure: deployment-mx has old puppetmaster stuff - https://phabricator.wikimedia.org/T152353#2847868 (10Krenair) I'd kind of prefer to replace the instance. I don't like instances with unpuppetised stuff sitting around. [18:02:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 06Services (watching): Move primary trending service development to github - https://phabricator.wikimedia.org/T151469#2847874 (10greg) Should we re-title/focus this task or decline this one and make a new... [18:02:54] (03PS1) 10Hashar: Add trendingedits packages to CI image [integration/config] - 10https://gerrit.wikimedia.org/r/325330 (https://phabricator.wikimedia.org/T151469) [18:03:36] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 13Patch-For-Review, 06Services (watching): Move primary trending service development to github - https://phabricator.wikimedia.org/T151469#2847882 (10Ottomata) Related: https://phabricator.wikimedia.org/T... [18:06:52] 10Continuous-Integration-Config, 06Discovery, 10Wikimedia-Portals: wikimedia/portals repo might be using outdated and or deprecated tests in jenkins - https://phabricator.wikimedia.org/T152351#2847903 (10MarcoAurelio) After merging some code, it is now: ``` 17:21:46 [wikimedia-portals-npm-node-4-jessie] $ /... [18:09:34] 10Continuous-Integration-Config, 06Discovery, 10Wikimedia-Portals: wikimedia/portals repo might be using outdated and or deprecated tests in jenkins - https://phabricator.wikimedia.org/T152351#2847915 (10hashar) ``` Python executable "C:\Users\MA\AppData\Local\... [18:09:50] 10Continuous-Integration-Config, 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream: Improve tests for KafkaSSE - https://phabricator.wikimedia.org/T150436#2847916 (10hashar) [18:17:00] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [18:17:38] 10Beta-Cluster-Infrastructure, 10Shinken, 13Patch-For-Review, 07Wikimedia-Incident, 07Wikimedia-log-errors: Shinken alert for beta error rate - https://phabricator.wikimedia.org/T141785#2847954 (10Krenair) 05Open>03Resolved a:03thcipriani http://shinken.wmflabs.org/service/graphite-labs/Mediawiki%2... [18:23:36] 10Continuous-Integration-Config, 06Discovery, 10Wikimedia-Portals: wikimedia/portals repo might be using outdated and or deprecated tests in jenkins - https://phabricator.wikimedia.org/T152351#2847998 (10Jdrewniak) @MarcoAurelio this looks like an error compiling `lwip` - the node image processor & optimizer... [18:28:29] Project beta-update-databases-eqiad build #13303: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13303/ [18:45:03] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team: Make it possible to execute tests as a specific MediaWiki user on beta cluster - https://phabricator.wikimedia.org/T152432#2848071 (10Jdlrobson) [18:46:26] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team: Make it possible to execute tests as a specific (new) MediaWiki user on beta cluster - https://phabricator.wikimedia.org/T152432#2848018 (10Jdlrobson) [18:53:36] Yippee, build fixed! [18:53:37] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #251: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/251/ [18:59:06] 10Gerrit, 13Patch-For-Review, 07Upstream: Free-form tagging in gerrit - https://phabricator.wikimedia.org/T37534#2848153 (10Paladox) This now works as the above commit was merged. [19:01:38] Yippee, build fixed! [19:01:39] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #251: 09FIXED in 30 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/251/ [19:13:29] Project beta-update-databases-eqiad build #13304: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13304/ [19:14:41] twentyafterfour hi, upstream seem to have done more work on ipv6 support, https://secure.phabricator.com/rPHU2ffd89902c1b6fccd71f135eda479d19bc558646 [19:14:53] https://secure.phabricator.com/D16986 [19:15:03] https://secure.phabricator.com/D16987 [19:15:16] mutante ^^ so looks like ipv6 support is almost fixed [19:18:57] paladox: yep looks good [19:19:06] yep :) [19:19:17] paladox: sounds cool [19:19:23] :) [19:20:27] that looks like enough to make it work instead of blow up like it did last week [19:21:20] i wonder if search.svc speaks https [19:21:22] looks [19:21:34] it's just because now whenever i see a http URL it triggers me :p [19:21:49] mutante: yeah ideally we should use https if it's available [19:21:59] since phabricator search can return confidential info [19:23:09] 06Release-Engineering-Team, 10ChangeProp, 06Operations, 06Parsing-Team, and 5 others: Separate clusters for asynchronous processing from the ones for public consumption - https://phabricator.wikimedia.org/T152074#2848311 (10GWicke) There are pros & cons for dividing the API cluster in multiple sub-clusters... [19:23:30] yep [19:23:57] twentyafterfour will there be a phab update this week? We could backport those ipv6 changes into the update? [19:24:07] is there any kind of known issue with tests for wmf.4? [19:24:35] asking because https://gerrit.wikimedia.org/r/#/c/325246/ fails php tests but is a javascript change? [19:24:38] well, here's a sign that it may work [19:24:39] hieradata/role/codfw/elasticsearch/cirrus.yaml:elasticsearch::https::certificate_name: 'search.svc.codfw.wmnet' [19:24:45] https cert name search.. [19:25:12] 19:19:16 Warning: Destructor threw an object exception: exception 'DBTransactionError' with message 'LBFactory::shutdown: transaction round 'AtomicSectionUpdate::doUpdate' still running.' in /home/jenkins/workspace/mediawiki-extensions-hhvm-jessie/src/includes/libs/rdbms/lbfactory/LBFactory.php:208 [19:25:27] 19:19:16 Fatal error: Class undefined: Wikimedia\Rdbms\SessionConsistentConnectionManager in /home/jenkins/workspace/mediawiki-extensions-hhvm-jessie/src/extensions/Wikidata/extensions/Wikibase/client/includes/Store/Sql/DirectSqlStore.php on line 238 [19:25:32] thcipriani ^^ [19:25:48] that sounds like wmf5 will fix that [19:26:31] I see a few errors in that output, if tests aren't working for wmf.4 that'd be bad. [19:26:33] Similar to https://phabricator.wikimedia.org/T152357 [19:26:44] paladox: do you mean wmf.5 of...? core or [19:26:55] Yes [19:27:33] The change should be in core [19:27:38] we just need to backport it [19:28:45] thcipriani found it [19:28:51] It's missing https://github.com/wikimedia/mediawiki/blob/c02d1fb4c5aaa57f2c33e39dc4753b8111a62d53/includes/libs/rdbms/connectionmanager/SessionConsistentConnectionManager.php [19:29:19] but it has a few commits so i am unsure how to back port it [19:31:24] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 10MediaWiki-Unit-tests, 13Patch-For-Review: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.... - https://phabricator.wikimedia.org/T137460#2848327 [19:31:28] twentyafterfour: paladox : it looks to me like we have https::certificate_name set for search.svc but besides that it may not be setup or not in puppet yeat, but not sure [19:31:37] Oh [19:31:57] i tried to find if it's port 9201 or something [19:32:09] oh [19:32:24] cant just change protocol and not change port [19:33:03] can you see anything else that indicates https on search.svc paladox? [19:33:18] Not sure. let me check [19:34:26] mutante mediawiki is https according to https://github.com/wikimedia/operations-puppet/search?utf8=%E2%9C%93&q=https%3A%2F%2Fsearch.svc.eqiad.wmnet%3A9200&type=Code [19:36:27] mutante the only way to test is if we try it on the phabricator instance [19:36:53] paladox: got the answer from 11:35 < ebernhardson> mutante: 9243 [19:36:57] Oh [19:37:01] :) [19:37:44] if you made it a few lines further down in that search query it said 9243 as well ;) [19:37:51] (not the query, but the file it points to) [19:37:57] Oh [19:38:34] twentyafterfour: let's try it with https:// :9243 [19:48:08] twentyafterfour upstream seem to have fixed most of the ipv6 problems, inbound works but outbound does not work yet [19:48:09] see https://secure.phabricator.com/T11939#202587 [19:50:21] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 10MediaWiki-Unit-tests, and 2 others: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedi... - https://phabricator.wikimedia.org/T137460#2848414 [20:05:00] Project beta-update-databases-eqiad build #13305: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13305/ [20:30:43] (03CR) 10Hashar: [C: 032] Add trendingedits packages to CI image [integration/config] - 10https://gerrit.wikimedia.org/r/325330 (https://phabricator.wikimedia.org/T151469) (owner: 10Hashar) [20:31:48] (03Merged) 10jenkins-bot: Add trendingedits packages to CI image [integration/config] - 10https://gerrit.wikimedia.org/r/325330 (https://phabricator.wikimedia.org/T151469) (owner: 10Hashar) [20:34:04] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 13Patch-For-Review, 06Services (watching): Move primary trending service development to github - https://phabricator.wikimedia.org/T151469#2848633 (10Nuria) If I am the only one that thinks this way fee... [20:43:22] !log Image ci-jessie-wikimedia-1480969940 in wmflabs-eqiad is ready (include trendingedits::packages which explicitly define the installation of librdkafka-dev' ) [20:43:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:05:01] Project beta-update-databases-eqiad build #13306: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13306/ [21:11:52] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, 13Patch-For-Review, 06Services (watching): Move primary trending service development to github - https://phabricator.wikimedia.org/T151469#2848768 (10hashar) I have refreshed the CI image. Turns out that... [21:21:10] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream: Improve tests for KafkaSSE - https://phabricator.wikimedia.org/T150436#2848823 (10hashar) I am merely brain dumping my understanding for this task: node-rdkafka are nodejs bindings for librdk... [21:29:39] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup trending service CI - https://phabricator.wikimedia.org/T151469#2848856 (10greg) p:05Triage>03Normal [21:41:07] hashar: any idea what's wrong with https://integration.wikimedia.org/ci/job/mwext-qunit-jessie/6571/console ? seems like it's running in the wrong language (mw.language.convertNumber uses a comma as decimal mark) [21:43:24] tgr: I recently change how mw.language.convertNumber handles conversion to integer, might be related [21:43:24] hashar: unrelated: I recently created integration tests for core in https://gerrit.wikimedia.org/r/#/c/318658/ not sure if those should be included in CI somehow [21:44:17] Nikerabbit: the error is [21:44:18] 19:37:34 Expected: "Location: 12° 20′ 44.44″ N, 98° 45′ 55.56″ E" [21:44:21] 19:37:34 Actual: "Location: 12° 20′ 44,44″ N, 98° 45′ 55,56″ E" [21:44:44] and the test itself does not do anything with languages [21:46:02] so not related to integers as fas as Ican see [21:47:52] k [22:05:00] Project beta-update-databases-eqiad build #13307: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13307/ [22:14:29] 06Release-Engineering-Team, 10ChangeProp, 06Operations, 06Parsing-Team, and 5 others: Separate clusters for asynchronous processing from the ones for public consumption - https://phabricator.wikimedia.org/T152074#2849092 (10greg) @Joe Should I make this an explicit follow-up from the incident? https://wiki... [23:00:27] RECOVERY - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is OK: OK: Less than 100.00% above the threshold [0.0] [23:05:00] Project beta-update-databases-eqiad build #13308: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/13308/ [23:09:12] tgr: sorry havent seen your note :/ [23:09:32] tgr: and I am really heading to bed. Feel free to copy paste to a Phab task and I will follow up [23:10:11] * hashar waves [23:49:28] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:53:24] 06Release-Engineering-Team, 06Collaboration-Team-Triage, 10Flow, 07Beta-Cluster-reproducible: Beta update.php fails:The content model 'CONTENT_MODEL_FLOW_BOARD' is not registered on this wiki. - https://phabricator.wikimedia.org/T152379#2849397 (10Catrope) >>! In T152379#2846363, @hashar wrote: > A couple... [23:54:01] deployment-logstash2 is because we moved the logstash class [23:54:09] double checked everything in prod [23:54:52] 03Scap3: ElectronPdfService mw extension l10n messages missing after full scap sync - https://phabricator.wikimedia.org/T152424#2849401 (10thcipriani) p:05Triage>03High Hrm. So in doing some digging, it seems like the electronpdf service is missing from extensionmessages: ``` grep -i electron /srv/mediawiki... [23:57:27] bd808: or greg-g are you around for a quick change in wikitech ui? [23:58:00] mutante: what's up? [23:58:18] bd808: i amended and merged your change about splitting logstash roles [23:58:22] we need to reconfigure the instance [23:58:25] deployment-logstash2 [23:58:27] ah. ok [23:58:32] to use the new role name [23:58:35] logstash::collector [23:59:00] * bd808 pokes around in horizon [23:59:28] maybe it's still the old wikitech ui [23:59:30] and "configure" [23:59:42] nope. we moved all that to horizon [23:59:50] then it's good that i asked