[00:36:03] 10Release-Engineering-Team (Kanban), 10User-greg: Distill OCG background into retrospective etherpad - https://phabricator.wikimedia.org/T174735#3574593 (10greg) First pass done. Going to pass over to others for input. [00:53:52] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T170636#3574598 (10greg) a:03mmodell [00:54:03] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T170637#3574600 (10greg) a:03mmodell [04:03:37] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #504: 04FAILURE in 7 min 36 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/504/ [04:24:51] Yippee, build fixed! [04:24:52] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #504: 09FIXED in 28 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/504/ [05:17:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:47:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:21:18] 10Release-Engineering-Team (Watching / External), 10Wikidata: Document wikidata deployment properly - https://phabricator.wikimedia.org/T168491#3574697 (10eranroz) [06:21:20] 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#3574698 (10eranroz) [07:19:44] 10Gerrit, 10Repository-Ownership-Requests, 10VPS-project-libraryupgrader: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574757 (10Legoktm) >>! In T174760#3573778, @demon wrote: > But also....if we're using the API we're having to auth using the actual LDA... [07:40:07] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #471: 15ABORTED in 3 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/471/ [07:40:08] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #471: 15ABORTED in 3 hr 0 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/471/ [07:52:24] 10Gerrit, 10Repository-Ownership-Requests, 10VPS-project-libraryupgrader: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574763 (10Paladox) If your using HTTP digest authentication, note that this was removed with gerrit 2.14. [07:57:14] 10Gerrit, 10Repository-Ownership-Requests, 10VPS-project-libraryupgrader: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574764 (10Legoktm) >>! In T174760#3574763, @Paladox wrote: > If your using HTTP digest authentication, note that this was removed with... [08:10:03] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [08:32:49] !log rm -rf /var/logs/kafka on deployment-kafka01 to free up disk space [08:32:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:33:59] 10Release-Engineering-Team (Kanban), 10User-greg: 201718Q2 RelEng related program goals - https://phabricator.wikimedia.org/T174835#3574780 (10Aklapper) [08:35:04] 10Beta-Cluster-Infrastructure, 10Analytics: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3574783 (10Legoktm) p:05Triage>03High I just ran rm -rf /var/log/kafka to free up a little bit of space, but this needs attention from someone who understands kafka. [08:42:21] 10Gerrit, 10Repository-Ownership-Requests, 10VPS-project-libraryupgrader: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574788 (10Paladox) @Legoktm see https://gerrit-review.googlesource.com/Documentation/rest-api.html#authentication (this is from the mas... [08:44:02] 10Gerrit, 10Repository-Ownership-Requests, 10VPS-project-libraryupgrader: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574801 (10Legoktm) Ah they switched to HTTP Basic Auth. That'll be trivial to migrate to in the future. [08:46:05] 10Gerrit, 10Repository-Ownership-Requests, 10VPS-project-libraryupgrader: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574802 (10Paladox) HTTP basic authentication is recommended now. [09:30:04] RECOVERY - Free space - all mounts on deployment-kafka01 is OK: OK: All targets OK [09:58:29] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3574876 (10Addshore) [12:22:52] Yippee, build fixed! [12:22:52] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #513: 09FIXED in 51 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/513/ [13:46:02] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #510: 04FAILURE in 2 min 1 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/510/ [14:16:18] (03PS1) 10Umherirrender: Use composer unit tests for some extensions [integration/config] - 10https://gerrit.wikimedia.org/r/375527 [16:01:13] (03CR) 10Samtar: [C: 04-1] "Missing end of string assertion" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/375428 (owner: 10Zppix) [16:15:01] (03PS3) 10Zppix: add samtar to ci whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/375428 [16:15:13] (03CR) 10Zppix: "Good catch Samtar" [integration/config] - 10https://gerrit.wikimedia.org/r/375428 (owner: 10Zppix) [16:16:30] (03CR) 10Samtar: [C: 031] add samtar to ci whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/375428 (owner: 10Zppix) [17:40:49] krenair@shinken-01:~$ /usr/lib/nagios/plugins/check_graphite -U https://graphite-labs.wikimedia.org -T 10 check_series_threshold 'deployment-prep.deployment-tin.diskspace.*.byte_percentfree' -W 15 -C 10 --from 10min --until 0min --perc 1 --under --allow-undefined [17:40:49] WARNING: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<100.00%) [17:42:25] there is no /mnt on deployment-tin [17:43:44] so deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree must exist somehow but not have valid datapoints? [17:57:10] looks like all datapoints are None which fails https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/nagios_common/files/check_commands/check_graphite;3a5e8f475a1cb80b7759e7bb0be353a9dacbc60b$324 [17:57:29] I wonder if we should check self.allow_undefined before that continue [18:00:25] no that's not quite right [19:10:14] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T170634#3575267 (10aaron) [19:10:16] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T170632#3575268 (10aaron) [19:21:04] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<11.11%) [19:56:03] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<11.11%) [20:41:47] Yippee, build fixed! [20:41:48] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #505: 09FIXED in 47 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/505/ [20:41:51] Yippee, build fixed! [20:41:51] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #505: 09FIXED in 50 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/505/ [22:21:33] Yippee, build fixed! [22:21:34] Project selenium-CentralAuth » firefox,beta,Linux,BrowserTests build #507: 09FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/507/