[00:10:09] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362757 (10Paladox) @mmodell thanks. [01:32:59] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362934 (10Dzahn) i setup a labs environment to test this as much as we like without needing prod see this: http://... [01:40:15] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2362935 (10Dzahn) This works for the first row in the table: ``` RewriteCond %{HTTP_HOST} =git.wikimedia.org Rew... [02:07:32] 06Release-Engineering-Team, 06Developer-Relations, 06Team-Practices: Setup a codereview hour for non american people - https://phabricator.wikimedia.org/T136370#2362942 (10mmodell) 05Open>03Resolved a:03mmodell [02:46:00] eh wtf [02:46:05] gearman stuck maybe? [02:47:54] !log disabling/enabling gearman in jenkins because everything is stuck [02:48:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:48:01] java.io.IOException: Failed to create a temporary file in /var/lib/jenkins [02:49:51] legoktm@gallium:/var/lib/jenkins$ touch test [02:49:51] touch: cannot touch `test': Read-only file system [02:56:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2362975 (10Legoktm) [02:56:39] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2362987 (10Legoktm) p:05Triage>03Unbreak! [02:56:56] !log / on gallium is read-only [02:57:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:57:51] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2362989 (10yuvipanda) I see an mdm alert for it: ``` This is an automatically generated mail message from mdadm running on gallium A Fai... [03:51:10] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363081 (10yuvipanda) fsck completed with: ``` root@gallium:/home/yuvipanda# fsck.ext3 -n /dev/md0 | tee fsck tee: fsck: Read-only file s... [04:10:59] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363110 (10yuvipanda) I suspect rebooting + fsck on reboot will fix this, but I'm also aware that I haven't done this before, and that gal... [04:59:02] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2362975 (10MoritzMuehlenhoff) mdadm shows /dev/sda2 as failed, so it needs to be removed from /dev/md0 and replaced. Let's wait for Antoin... [05:28:02] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [07:11:00] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2362975 (10Joe) Please don't reboot the machine: while `/dev/sda2` seems to be failing, we also have `/dev/sdc` reporting I/O errors ```... [07:33:57] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363275 (10Joe) ``` mdadm --detail /dev/md0 /dev/md0: Version : 0.90 Creation Time : Thu Aug 25 21:30:22 2011 Raid Level :... [07:42:32] oh shit :( [07:58:25] ACKNOWLEDGEMENT - MD RAID on gallium is CRITICAL: CRITICAL: Active: 1, Working: 1, Failed: 1, Spare: 0 Giuseppe Lavagetto T137265 [07:58:25] ACKNOWLEDGEMENT - puppet last run on gallium is CRITICAL: CRITICAL: Puppet last ran 8 hours ago Giuseppe Lavagetto T137265 [08:11:58] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2363305 (10Paladox) Thanks :) [08:19:58] (03CR) 10JanZerebecki: "Would +2 but CI is down." [integration/config] - 10https://gerrit.wikimedia.org/r/293235 (owner: 10Ladsgroup) [08:33:01] PROBLEM - zuul_gearman_service on gallium is CRITICAL: Connection refused [08:33:42] PROBLEM - zuul_merger_service_running on gallium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [08:34:02] PROBLEM - zuul_service_running on gallium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-server [08:36:35] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363349 (10hashar) Entirely my fault for not having prepared a proper backup of gallium T80385 and not having moved gallium to another hos... [08:38:42] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363351 (10Joe) I think our best bet at the moment is installing a new system to replace gallium. @hashar suggested moving to jessie dir... [08:38:42] PROBLEM - jenkins_service_running on gallium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/java .*-jar /usr/share/jenkins/jenkins.war [08:39:02] PROBLEM - jenkins_zmq_publisher on gallium is CRITICAL: Connection refused [08:42:09] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363358 (10Joe) I don't have rights to edit the spares allocation spreadsheet, so I can't comment there, but I am thinking of allocating `... [08:50:37] (03PS1) 10Hashar: Merge commit '8c250cf' into debian/jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/293269 [09:07:10] 06Release-Engineering-Team, 10Phabricator: Create a notice panel on phabricator homepage - https://phabricator.wikimedia.org/T137278#2363392 (10Paladox) [09:07:34] 06Release-Engineering-Team, 06Operations, 10Phabricator: Create a notice panel on phabricator homepage - https://phabricator.wikimedia.org/T137278#2363407 (10Paladox) [09:08:05] 06Release-Engineering-Team, 06Operations, 10Phabricator: Create a notice panel on phabricator homepage - https://phabricator.wikimedia.org/T137278#2363392 (10Paladox) @mmodell or @Aklapper would you be able to do it please. [09:08:18] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: Port Zuul package 2.1.0-95-g66c8e52 from Precise to Jessie - https://phabricator.wikimedia.org/T137279#2363409 (10hashar) [09:08:40] (03PS2) 10Hashar: Merge commit '8c250cf' into debian/jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/293269 (https://phabricator.wikimedia.org/T137279) [09:09:20] andre__ Hi [09:09:24] could you do https://phabricator.wikimedia.org/T137278 please [09:09:45] 06Release-Engineering-Team, 06Operations, 10Phabricator: Create a notice panel on phabricator homepage - https://phabricator.wikimedia.org/T137278#2363427 (10Paladox) Seems we should create a new project that we lock down to only users can add only users instead of freely allowing joining. But could the rele... [09:09:48] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Port Zuul package 2.1.0-95-g66c8e52 from Precise to Jessie - https://phabricator.wikimedia.org/T137279#2363428 (10hashar) I have spawned a Jessie labs instance `zuul-dev-jessie.integration.eqiad.wmflabs`... [09:16:10] 06Release-Engineering-Team, 06Operations, 10Phabricator: Create a notice panel on phabricator homepage - https://phabricator.wikimedia.org/T137278#2363392 (10Peachey88) If they don't read the relevant mailing lists, what makes you think they will read the front page of phabricator? [09:17:32] 06Release-Engineering-Team, 06Operations, 10Phabricator: Create a notice panel on phabricator homepage - https://phabricator.wikimedia.org/T137278#2363461 (10JanZerebecki) I got to it to create a new ticket. Is there a way to add a notice above the create ticket form without being able to edit the rest of th... [09:17:40] 06Release-Engineering-Team, 06Operations, 10Phabricator: Create a notice panel on phabricator homepage - https://phabricator.wikimedia.org/T137278#2363462 (10Paladox) Since they have to create a task like T137276 did. [09:18:52] 06Release-Engineering-Team, 06Operations, 10Phabricator: Create a notice panel on phabricator homepage - https://phabricator.wikimedia.org/T137278#2363463 (10Paladox) @JanZerebecki https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/1/ [09:28:44] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363483 (10hashar) I am rebuilding/testing the Zuul deb package for Jessie (T137279). I have created a placeholder incident report on htt... [09:28:50] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T136042#2363485 (10Glaisher) [09:36:52] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363502 (10Joe) The host I chose was already allocated to maps100*, so we are now targeting `wmf4746` instead. [09:38:22] (03CR) 10Paladox: [C: 031] Merge commit '8c250cf' into debian/jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/293269 (https://phabricator.wikimedia.org/T137279) (owner: 10Hashar) [09:55:15] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T136042#2363544 (10Tgr) [09:58:29] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363564 (10Joe) smartclt status for both disks: - sdc P3220 - sda P3221 [10:02:44] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T136042#2363587 (10dcausse) [10:04:22] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T136042#2363594 (10aaron) [10:21:37] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, 13Patch-For-Review: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2363621 (10Ottomata) Hm, interesting, ja will look into how this is working in prod in the first place. @mobrovac, for the future,... [10:21:52] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Port Zuul package 2.1.0-95-g66c8e52 from Precise to Jessie - https://phabricator.wikimedia.org/T137279#2363622 (10hashar) @MoritzMuehlenhoff is taking care of adding the Jenkins 1.652.2 Debian packages fo... [10:36:04] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2363649 (10hashar) @MoritzMuehlenhoff is taking care of adding the Jenkins 1.652.2 Debian packages for jessie-wikime... [10:37:33] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Port Zuul package 2.1.0-95-g66c8e52 from Precise to Jessie - https://phabricator.wikimedia.org/T137279#2363665 (10hashar) p:05High>03Normal The package seems to work fine. I had a dummy zuul layout a... [10:51:50] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2363735 (10Paladox) [10:57:56] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2363747 (10Paladox) [11:10:46] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: doc.wikimedia.org should be running PHP 5.5+, not 5.3 -> demos etc. don't work - https://phabricator.wikimedia.org/T127504#2363758 (10Paladox) gallium had developed a fault today so they are creating a new instance running Jessie which will r... [11:10:54] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2362975 (10Paladox) [11:10:56] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: doc.wikimedia.org should be running PHP 5.5+, not 5.3 -> demos etc. don't work - https://phabricator.wikimedia.org/T127504#2363760 (10Paladox) [11:16:31] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: doc.wikimedia.org should be running PHP 5.5+, not 5.3 -> demos etc. don't work - https://phabricator.wikimedia.org/T127504#2363774 (10Paladox) https://gerrit.wikimedia.org/r/#/c/293284/ [12:08:55] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Update all references to gallium and change it to contint1001 in integration/* - https://phabricator.wikimedia.org/T137293#2363992 (10Paladox) [12:09:06] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Update all references to gallium and change it to contint1001 in integration/* - https://phabricator.wikimedia.org/T137293#2364006 (10Paladox) [12:09:08] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2362975 (10Paladox) [12:12:04] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Update all references to gallium and change it to contint1001 in integration/* - https://phabricator.wikimedia.org/T137293#2364029 (10Paladox) [12:51:17] (03PS1) 10Hashar: gallium is replaced by contint1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137265) [12:51:50] 10Deployment-Systems, 10Flow: Message Mediawiki:Flow-terms-of-use-edit on nowp is English - https://phabricator.wikimedia.org/T133571#2236107 (10Nemo_bis) [12:52:08] RECOVERY - MD RAID on gallium is OK: OK: Active: 1, Working: 1, Failed: 0, Spare: 0 [12:53:00] (03PS2) 10Paladox: gallium is replaced by contint1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137265) (owner: 10Hashar) [12:53:20] (03CR) 10Paladox: [C: 031] gallium is replaced by contint1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137265) (owner: 10Hashar) [13:30:09] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2364339 (10hashar) [13:31:25] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2364341 (10Paladox) [13:33:08] 10releng-201516-q3, 10Continuous-Integration-Infrastructure (phase-out-gallium), 07Jenkins: [keyresult] Migrate Jenkins to Jessie (gallium -> cobalt) - https://phabricator.wikimedia.org/T124121#2364350 (10Paladox) [14:05:23] PROBLEM - jenkins_zmq_publisher on contint1001 is CRITICAL: connect to address 127.0.0.1 and port 8888: Connection refused [14:10:46] hashar hi should we do https://gerrit.wikimedia.org/r/#/c/289451/ again [14:10:52] a site notice on integration [14:11:27] please [14:15:47] (03PS1) 10Paladox: zuul status: notice about ongoing outage [integration/docroot] - 10https://gerrit.wikimedia.org/r/293313 (https://phabricator.wikimedia.org/T137265) [14:18:28] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2364497 (10Paladox) [14:38:05] paladox: the machine has read only disks so... nop [14:38:07] can't update it [14:38:20] hashar oh [14:38:42] hashar now once the new server is up [14:38:50] we can migrate more tests from precise. [14:39:19] that is unrelated [14:39:27] no jobs are running on the server that die [14:39:34] (03Abandoned) 10Paladox: zuul status: notice about ongoing outage [integration/docroot] - 10https://gerrit.wikimedia.org/r/293313 (https://phabricator.wikimedia.org/T137265) (owner: 10Paladox) [14:39:35] and the remaining precise jobs are for Zend 5.3 [14:39:37] :D [14:39:46] hashar oh, but integration test [14:39:50] can be migrated [14:39:56] (03CR) 10Krinkle: zuul status: notice about ongoing outage (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/293313 (https://phabricator.wikimedia.org/T137265) (owner: 10Paladox) [14:40:03] yeah got a patch for that already [14:41:53] (03CR) 10Hashar: "Note gallium partitions are mounted read only due to disk errors. So we can not update the Zuul status page. Given it shows:" [integration/docroot] - 10https://gerrit.wikimedia.org/r/293313 (https://phabricator.wikimedia.org/T137265) (owner: 10Paladox) [14:44:57] hashar Ok :) [14:44:59] thanks [14:47:18] hashar i mean these jobs [14:47:20] - integration-jjb-config-diff [14:47:20] - integration-zuul-layoutdiff [14:47:20] - integration-zuul-layoutvalidation [14:49:23] paladox: yeah I have a patch for them [14:49:56] hashar ok thanks :) [14:50:14] hashar would us migrating require gerrit too since it is running on precise [14:51:57] hashar is it this patch https://phabricator.wikimedia.org/rCICF7d110aab9c5227e5b6c963ad97b748fb335ac96a [14:52:20] paladox: I am in the middle of rebuilding gallium / the whole CI infra, so I can't really look at that [14:52:35] need jenkins/zuul to be back [14:52:39] the ci config can wait really [14:52:45] hashar ok [15:11:47] 10Continuous-Integration-Infrastructure, 07Upstream: mediawiki/extensions.git does not update some extensions - https://phabricator.wikimedia.org/T51846#2364694 (10Paladox) Fixed in https://gerrit-review.googlesource.com/#/c/69891/ Upgrading to gerrit 2.12 which will happen soon will fix the problem. [15:14:14] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2364695 (10hashar) @joe got a new server, did a nice partition schema based on lvm. Had to poli... [15:32:08] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2364702 (10hashar) Status: @jcrespo has taken backups and is dealing with the disk failure + RA... [15:35:49] PROBLEM - zuul_gearman_service on contint1001 is CRITICAL: connect to address 127.0.0.1 and port 4730: Connection refused [15:36:08] PROBLEM - zuul_service_running on contint1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-server [15:36:31] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, 13Patch-For-Review: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2364721 (10thcipriani) >>! In T137192#2363621, @Ottomata wrote: > Hm, interesting, ja will look into how this is working in prod in... [15:36:50] hashar with https://gerrit.wikimedia.org/r/#/c/293283/ you said pending restoration of service of gallium [15:37:10] im wondering what you mean by that. [15:37:38] 10Beta-Cluster-Infrastructure, 03Scap3, 10EventBus, 06Services, 13Patch-For-Review: SSH key issue when deploying eventbus in Beta - https://phabricator.wikimedia.org/T137192#2364725 (10Ottomata) Updated patch: https://gerrit.wikimedia.org/r/#/c/293217/ [15:37:48] Are we going to the new server or are we going to see if we can restore gallium and if not go to the new server. [15:38:05] paladox: no idea yet [15:38:15] paladox: hopefully have the disk restored on gallium [15:38:19] else outage will continue till contint1001 is ready [15:38:22] hashar Ok [15:38:48] the low level hardware works happens on a private irc channel [15:38:58] Oh ok [15:39:50] Thanks for replying [15:46:57] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2364753 (10jcrespo) It seems as if the RAID operations were successful, but it got stuck on boot... [15:49:09] ACKNOWLEDGEMENT - jenkins_zmq_publisher on contint1001 is CRITICAL: connect to address 127.0.0.1 and port 8888: Connection refused Giuseppe Lavagetto Work in progress installation. [15:49:09] ACKNOWLEDGEMENT - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 1 failures Giuseppe Lavagetto Work in progress installation. [15:49:10] ACKNOWLEDGEMENT - zuul_gearman_service on contint1001 is CRITICAL: connect to address 127.0.0.1 and port 4730: Connection refused Giuseppe Lavagetto Work in progress installation. [15:49:11] ACKNOWLEDGEMENT - zuul_service_running on contint1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-server Giuseppe Lavagetto Work in progress installation. [16:04:52] 03Scap3, 10scap: Need a way to restart services without deploying via scap - https://phabricator.wikimedia.org/T119449#2364838 (10thcipriani) 05Open>03Resolved a:03thcipriani We have that now with `scap deploy --service-restart` [16:08:44] RECOVERY - zuul_merger_service_running on gallium is OK: PROCS OK: 1 process with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [16:08:54] RECOVERY - zuul_service_running on gallium is OK: PROCS OK: 2 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-server [16:09:32] hashar zuul is backonline https://integration.wikimedia.org/zuul/ [16:09:33] :) [16:11:42] hashar as gallium is will we still migrate to the new server or is it not a priority now. [16:12:02] is jenkins still down? [16:12:24] I want this to be merged https://gerrit.wikimedia.org/r/#/c/293235/ [16:12:24] Amir1 yes but looks like it is starting up now [16:12:30] it would be great [16:12:36] paladox: thanks :) [16:12:38] Amir1 you may have to v+2. [16:12:54] Your welcome [16:13:05] Amir1 https://integration.wikimedia.org/ci/ [16:13:33] I don't have enough rights to +2 the above patch, it's about zuul actually [16:13:54] 10Deployment-Systems, 03Scap3: puppet should run`deploy-init` command once after cloning the deploy repo - https://phabricator.wikimedia.org/T129906#2364865 (10mmodell) [16:13:55] Amir1 oh, looks like zuul gone down again [16:14:20] oh [16:14:24] 10Deployment-Systems, 03Scap3: puppet should run`deploy-init` command once after cloning the deploy repo - https://phabricator.wikimedia.org/T129906#2119521 (10mmodell) the `deploy --init` command exists but we still need to run it from puppet. [16:14:28] it can wait [16:14:34] Ok [16:15:59] 10Deployment-Systems, 03Scap3, 10scap: Scap3 checks.yaml should be environment specific - https://phabricator.wikimedia.org/T130558#2364870 (10thcipriani) 05Open>03Resolved a:03thcipriani This was resolved in {751496225cb7} should go out with v.3.2.1 [16:18:54] 10Deployment-Systems, 03Scap3: scap3 upstream/debian versioning - https://phabricator.wikimedia.org/T127828#2055121 (10thcipriani) Is there anything else that ought to be done here? [16:22:53] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2364905 (10hashar) [16:23:55] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2364922 (10hashar) We might need... [16:30:50] 10scap, 03Scap3 (Scap3-MediaWiki-MVP): Scap3 needs a way to handle large binary file transport - https://phabricator.wikimedia.org/T119443#2364926 (10mmodell) phabricator now supports git-lfs. [16:32:12] 10Deployment-Systems, 03Scap3, 06Operations, 13Patch-For-Review: Warning: rename(): Permission denied in /srv/mediawiki/wmf-config/CommonSettings.php on line 189 - https://phabricator.wikimedia.org/T136258#2328753 (10thcipriani) All patches for this task have merged. Anything left to do? [16:32:34] 10Deployment-Systems, 03Scap3: Make puppet runs of deploy-local more configurable - https://phabricator.wikimedia.org/T131627#2364936 (10mmodell) I thought deploy-local was only supposed to run once, not every puppet run. [16:37:44] 10Deployment-Systems, 03Scap3: Make puppet runs of deploy-local more configurable - https://phabricator.wikimedia.org/T131627#2364951 (10Ladsgroup) It tries to run deploy-local when Package[foo/deploy] is not there. And when that puppet fails (which can happen due to number of reasons) it tries to run deploy-l... [16:43:13] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2364976 (10hashar) The RAID array is rebuilding on gallium, would take ~1 hour and half. Puppet... [16:47:47] 10Deployment-Systems, 03Scap3: Make puppet runs of deploy-local more configurable - https://phabricator.wikimedia.org/T131627#2365015 (10mmodell) 05Open>03Resolved a:03mmodell @ladsgroup: Can you provide more details? AFAIK this is status:invalid [16:48:12] 10Deployment-Systems, 03Scap3: Make puppet runs of deploy-local more configurable - https://phabricator.wikimedia.org/T131627#2365020 (10mmodell) 05Resolved>03Open whoops. didn't mean to set resolved [17:08:40] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2365079 (10jcrespo) More details before I go: there are several backups on `db1085:/srv/backup/... [17:20:05] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2364905 (10mark) If I read this... [17:55:21] (03PS1) 10Cdentinger: Make DonationInterface test run against 1_26 branch. [integration/config] - 10https://gerrit.wikimedia.org/r/293344 [17:56:21] (03PS2) 10Cdentinger: WIP: Make DonationInterface test run against 1_26 branch. [integration/config] - 10https://gerrit.wikimedia.org/r/293344 [17:59:34] 10Deployment-Systems, 10Flow, 07I18n: Message Mediawiki:Flow-terms-of-use-edit on nowp is English - https://phabricator.wikimedia.org/T133571#2365254 (10Mattflaschen-WMF) [18:08:03] meanwhile gallium RAID rebuilding still has 40 mins to go ... [18:08:07] still away [18:11:16] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2365294 (10hashar) Sorry it is n... [18:14:05] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2365303 (10mmodell) ``` RewriteRule "^/tree/(.+).git" https://phabricator.wikimedia.org/r/p/%1;browse/ [R=301,NE] ``` [18:16:40] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2365308 (10mmodell) RewriteRule "^/log/(.+).git/refs/heads/(.*)" https://phabricator.wikimedia.org/r/p/%1;history/%2/... [18:19:17] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2365323 (10mmodell) ``` RewriteRule "^/commit/(.+)\.git/(\w+)" https://phabricator.wikimedia.org/r/revision/%1;%2 ``` [18:44:07] 10Beta-Cluster-Infrastructure, 10DBA, 10Flow, 03Collab-Team-2016-Apr-Jun-Q4: Run Flow External Store migration in dry-run mode on Beta - https://phabricator.wikimedia.org/T119567#2365401 (10Mattflaschen-WMF) Really sorry, I accidentally started running it without the dry run option. So it did it for real... [19:01:38] RECOVERY - jenkins_zmq_publisher on gallium is OK: TCP OK - 0.000 second response time on port 8888 [19:02:29] Amir1 zuul and jenkins are back online now. [19:03:36] 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: write Apache rewrite rules for gitblit -> diffusion migration - https://phabricator.wikimedia.org/T137224#2365441 (10Dzahn) here's another working one for the second type: https://gerrit.wikimedia.org/r/#/c/293221/4/module... [19:04:32] 10Beta-Cluster-Infrastructure, 10DBA, 10Flow, 03Collab-Team-2016-Apr-Jun-Q4: Run Flow External Store migration in dry-run mode on Beta - https://phabricator.wikimedia.org/T119567#2365447 (10Mattflaschen-WMF) Dry run completed: P3226 Skimming it, it looks good. @jcrespo Let me know when I should proceed t... [19:07:58] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2365455 (10hashar) [19:08:31] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2364905 (10hashar) I have split... [19:13:42] 10Continuous-Integration-Infrastructure, 10ArchCom-RfC, 13Patch-For-Review, 07RfC: [RFC] Optional Travis integration for Jenkins - https://phabricator.wikimedia.org/T114421#2365477 (10RobLa-WMF) p:05Triage>03Low Belated priority update discussed in {E187} (see log at P3179). > 21:41:50 T1144... [19:15:59] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2365483 (10hashar) 18:50 mark poked me stating that the raid rebuild is complete and gallium reb... [19:20:07] Project beta-update-databases-eqiad build #9030: 04FAILURE in 7.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9030/ [19:23:50] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2365518 (10hashar) [19:23:54] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2365519 (10hashar) [19:23:57] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: Port Zuul package 2.1.0-95-g66c8e52 from Precise to Jessie - https://phabricator.wikimedia.org/T137279#2365520 (10hashar) [19:24:43] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: / on gallium is read only, breaking jenkins - https://phabricator.wikimedia.org/T137265#2362975 (10hashar) 05Open>03Resolved gallium reboot apparently went with Zuul / Jenkins up s... [19:25:14] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: Port Zuul package 2.1.0-95-g66c8e52 from Precise to Jessie - https://phabricator.wikimedia.org/T137279#2363409 (10hashar) [19:28:25] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Nodepool can barely spawn instances OpenStack - https://phabricator.wikimedia.org/T137241#2365543 (10hashar) Thank you @Andrew for the details, happy you managed to figure it out. [19:30:59] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [19:53:17] 19:47:01 fatal: unable to access 'https://gerrit.wikimedia.org/r/p/mediawiki/extensions/AbuseFilter/': Could not resolve host: gerrit.wikimedia.org [19:53:23] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit-jessie/234/console - https://gerrit.wikimedia.org/r/#/c/293363/ [20:00:37] Krinkle works for me [20:00:54] paladox: What works for you? [20:01:03] Krinkle this https://gerrit.wikimedia.org/r/p/mediawiki/extensions/AbuseFilter/ [20:01:23] Yes, the problem is not with gerrit. [20:01:29] The problem is with Jenkins job slave. [20:01:43] Presumably its DNS or firewall [20:01:50] Krinkle oh maybe [20:02:15] Krinkle ci is migrating tomarror to a new server so lets hope that fixes the problems. [20:03:29] Hey, anyone with +2 rights in integration config can check this out? https://gerrit.wikimedia.org/r/#/c/293235/ [20:03:37] hashar: ^ it would be great, thanks [20:03:49] paladox: thanks for mentioning that jenkins is up now [20:03:58] writing an incident report right now [20:04:01] Your welcome [20:04:21] okay [20:04:23] Amir1 legoktm or jzerebecki have +2 but im not sure if there around [20:07:28] okay [20:15:28] (03CR) 10Krinkle: "Note: To avoid git log and production from diverging, per https://www.mediawiki.org/wiki/Continuous_integration/Zuul don't merge unless yo" [integration/config] - 10https://gerrit.wikimedia.org/r/293235 (owner: 10Ladsgroup) [20:19:31] Hi, is fawiki at group1 or group2? [20:19:46] Luke081515: it should group3 [20:19:50] it should be [20:19:53] 3? [20:20:03] we have 0, 1 and 2, free is new for me :D [20:20:04] s7, group3 (all Wikipedias) [20:20:05] Project beta-update-databases-eqiad build #9031: 04STILL FAILING in 4.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9031/ [20:20:18] hm, ok [20:20:20] I meant the third group [20:20:25] ok :D [20:20:27] counting starts from zero here :D [20:21:14] thx [20:26:50] ori or ostriches Hi, do you know if we have statistics on git.wikimedia.org please [20:27:16] paladox: I don't know [20:27:22] All we need to know is how much users visit it and how much users visit a certain link [20:27:24] Oh ok [20:27:29] mutante ^^ [20:29:43] paladox: Why? [20:30:18] Krinkle we are working on redirect links [20:30:49] Krinkle we wont know which links are the most used since we doint think we will be able to redirect every link to the correct place. [20:31:14] Krinkle https://git.wmflabs.org/ please [20:31:43] https://www.mediawiki.org/wiki/Special:LinkSearch/git.wikimedia.org [20:31:58] Krinkle thanks [20:32:13] https://www.mediawiki.org/w/index.php?title=Special:LinkSearch&limit=500&offset=0&target=http%3A%2F%2Fgit.wikimedia.org%2F [20:32:35] also [20:32:35] https://wikitech.wikimedia.org/w/index.php?title=Special:LinkSearch&limit=500&target=git.wikimedia.org%2F [20:32:38] For common links [20:32:51] Ok thanks [20:33:12] http://pix.toile-libre.org/upload/original/1465417955.png \o/ [20:33:38] the play by play for that image is :(, :X (dead), :) [20:35:07] Krinkle https://www.similarweb.com/website/git.wikimedia.org#referrals [20:48:37] 05Gitblit-Deprecate, 10Phabricator: Update all references to git.wikimedia.org and replace them with the phabricator equivilant - https://phabricator.wikimedia.org/T137353#2365874 (10Paladox) [20:49:40] Project beta-scap-eqiad build #105913: 04FAILURE in 4 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/105913/ [20:55:40] Yippee, build fixed! [20:55:40] Project beta-scap-eqiad build #105914: 09FIXED in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/105914/ [21:05:58] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T136042#2365927 (10thcipriani) [21:15:18] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2364905 (10hashar) [21:18:24] 06Release-Engineering-Team, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium), 13Patch-For-Review: Port Zuul package 2.1.0-95-g66c8e52 from Precise to Jessie - https://phabricator.wikimedia.org/T137279#2366001 (10hashar) [21:20:05] Yippee, build fixed! [21:20:05] Project beta-update-databases-eqiad build #9032: 09FIXED in 40 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/9032/ [21:23:55] (03PS3) 10Hashar: gallium is replaced by contint1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137293) [21:24:06] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Update all references to gallium and change it to contint1001 in integration/* - https://phabricator.wikimedia.org/T137293#2366091 (10hashar) Thank you for the task! [21:24:26] (03CR) 10Hashar: "I have dropped the task that dealt with replacing gallium disk." [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137293) (owner: 10Hashar) [21:24:47] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Update all references to gallium and change it to contint1001 in integration/* - https://phabricator.wikimedia.org/T137293#2366096 (10hashar) [21:24:49] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Update all references to gallium and change it to contint1001 in integration/* - https://phabricator.wikimedia.org/T137293#2366097 (10Paladox) Your welcome. [21:29:08] hashar are you starting the migration process tomarror for gallium. [21:30:09] paladox: maybe [21:30:20] paladox: it is not that trivial to do. We will see with ops tomorrow :) [21:30:26] hashar ok, thanks for replying. [21:30:27] :) [21:31:30] (03PS2) 10Hashar: Whitelist [lL]adsgroup [integration/config] - 10https://gerrit.wikimedia.org/r/293235 (owner: 10Ladsgroup) [21:33:29] (03CR) 10Hashar: [C: 032] "CI was down so yeah safer to hold the change in Gerrit in case it ends up having a nasty side effect :D" [integration/config] - 10https://gerrit.wikimedia.org/r/293235 (owner: 10Ladsgroup) [21:33:32] (03PS4) 10Paladox: gallium is replaced by contint1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137293) (owner: 10Hashar) [21:33:49] Amir1: updating your mail in Gerrit [21:33:49] (03CR) 10Paladox: "I Added task T137358 to the bug." [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137293) (owner: 10Hashar) [21:34:13] (03Merged) 10jenkins-bot: Whitelist [lL]adsgroup [integration/config] - 10https://gerrit.wikimedia.org/r/293235 (owner: 10Ladsgroup) [21:34:17] oh thanks [21:34:32] I'm sorry I had to put you into trouble [21:34:54] * Amir1 wishes he could be more helpful in zuul/CI [21:35:14] (03CR) 10Hashar: "T137358 Migrate CI services from gallium to contint1001" [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137293) (owner: 10Hashar) [21:35:22] (03PS5) 10Hashar: gallium is replaced by contint1001.eqiad.wmnet [integration/config] - 10https://gerrit.wikimedia.org/r/293300 (https://phabricator.wikimedia.org/T137293) [21:35:52] paladox: https://phabricator.wikimedia.org/T137293 is the more specific task :D it is already blocking the other so no need to reference both [21:36:07] hashar Oh, ok. [21:36:18] Amir1: well you already helped CI a lot! [21:36:35] Amir1: be it on pywikibot/core , making sure ORES has tests and associated jobs [21:37:04] thanks :) [21:37:05] Amir1: even figuring out a good strategy deployment for ORES (using wheels) is helping since CI has a couple python based software (Zuul and Nodepool) which we might want to deploy the same way [21:37:25] ^ analytics too! [21:37:30] wheels to the future! [21:37:30] hashar: We are testing git.wikimedia.org redirect links https://git.wmflabs.org/ [21:37:32] :) [21:37:33] hashar: we finally deployed ORES in prod via wheels [21:37:45] Amir1: I have deployed your change [21:37:52] ohhhhh [21:37:55] nice! thanks [21:38:04] mutante created those im helping him with what url should be converted to what. [21:38:09] not even in prod, the labs setup uses wheels as well [21:38:17] paladox: neat! :) [21:38:24] Yep :) [21:38:33] halfak: yeah wheels are quite a nice thing ;] [21:38:50] congratulations on your first wheels based deployment! [21:38:52] We are now working on getting https://git.wmflabs.org/log/mediawiki/core.git/refs%2Fheads%2Fmaster to work [21:39:04] hashar: maybe we can sit down and finish it for Zuul in Wikimania [21:39:35] Amir1: I am not attending :( [21:39:42] hashar but we are going to instead look at some stats about traffic who visited what the most since not all links will work. [21:39:43] :(((( [21:40:18] According to https://www.mediawiki.org/w/index.php?title=Special:LinkSearch&limit=500&offset=0&target=http%3A%2F%2Fgit.wikimedia.org now there is 246 links for git.wikimedia.org to be updated. [21:40:25] but we might well change the CI infra, need to find out new hosts for the various services on gallium [21:40:38] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T136042#2366134 (10thcipriani) [21:40:46] so maybe we would get everything on labs instance which would make it easier to enroll people in mainaining i [21:40:49] the task is roughly https://phabricator.wikimedia.org/T133300 [21:40:52] Amir1: ^ [21:41:18] let me take a look [21:43:14] Amir1: in short that is about migrating the various servinces on gallium to different places / hosts [21:43:21] maybe split them to individual machines / VM or whatever [21:43:36] in prod or labs? [21:43:42] no idea [21:43:49] have to sort that out with ops [21:44:12] there are several options, and I am bad at making choices [21:44:28] :D [21:44:56] Do you want to deprecate Zuul and use something else? [21:45:44] if not, I would suggest you to try using wheels before, What we did in ORES was using wheels in the labs setup (and beta cluster) and then moving to prod [21:46:00] Amir1 actually i like zuul [21:46:08] makes it easy to track tests [21:46:25] Even if we didn't end up migrating to prod we had a more robust and safe environment [21:47:02] paladox: I'm not saying we use wheels instead of Zuul, I'm saying we use wheels to deploy Zuul [21:47:22] Amir1 oh, im not sure what wheels is [21:47:36] Amir1: yeah I agree on labs -> prod that sounds like a good idea [21:47:47] and we might drop Zuul eventually [21:47:55] paladox: have you worked eggs (or virtualenvs) in python/ [21:48:05] who knows, we are not there yet ;) I am heading out to bed! [21:48:07] Amir1 nope [21:48:18] hashar: sleep tight o/ [21:48:34] paladox: python has its own system to install libraries [21:48:43] Oh [21:48:45] e.g. we do "pip install pywikibot" [21:48:54] and it installs pywikibot [21:49:25] it can do it via different methods, before wheels coming. it downloaded something called egg which was the source code [21:49:27] Oh yep [21:49:41] but wheels are binary containers [21:49:50] so we can ship them to other places [21:50:03] Ok [21:50:30] it's great for some python libraries that have c dependencies such as scipy, numpy , etc. [21:51:11] Yep [22:24:39] 10scap, 10Parsoid, 03Scap3 (Scap3-Adoption-Phase1): Deploy Parsoid with scap3 - https://phabricator.wikimedia.org/T120103#1844931 (10Jdforrester-WMF) Note that this is a class of tasks mentioned by RelEng in the SoS today; it's (one of several tasks) blocking the scap3 migration. [22:37:34] thcipriani: https://phabricator.wikimedia.org/T116340 seems poorly titled. [22:37:41] "Deploy Cassandra with scap3" [22:37:59] Don't we mean the 2 tools that plug into cassandra? Cassandra itself is a package afaik.... [22:38:07] (or am I missing something) [22:39:05] I *think* that is correct (deployed via package), but I'm not 100%; regardless, yes, it seems like the task is talking about a couple of specific tools there. [22:40:19] ah, it seems like the task description was edited [22:40:28] * thcipriani was wondering why it was only one task in the first place [22:43:34] my bad [22:49:39] 10scap, 10RESTBase-Cassandra, 10cassandra, 03Scap3 (Scap3-Adoption-Phase1): Deploy logstash logback encoder - https://phabricator.wikimedia.org/T116340#2366364 (10thcipriani) [22:50:24] 10releng-201516-q2, 10releng-201516-q3, 10scap, 03Scap3 (Scap3-Adoption-Phase1): [keyresult] Migrate all Service team owned services and MW to scap - https://phabricator.wikimedia.org/T109926#2366368 (10thcipriani) [22:50:33] ^ fixed :) [22:51:42] tyvm [22:52:12] thcipriani: now your math is wrong though! [22:52:29] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2364905 (10Dzahn) for the flows... [22:52:29] * greg-g goes [22:52:32] 10scap, 10RESTBase-Cassandra, 10cassandra, 03Scap3 (Scap3-Adoption-Phase1): Deploy logstash logback encoder with scap3 - https://phabricator.wikimedia.org/T116340#2366380 (10Legoktm) [22:54:40] (03PS3) 10Cdentinger: WIP: Make DonationInterface test run against 1_26 branch. [integration/config] - 10https://gerrit.wikimedia.org/r/293344 (https://phabricator.wikimedia.org/T137213) [22:55:54] eh, based my math off a directory listing, also: I stand by the "about half" statement [22:57:05] :) [22:57:07] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, 10Continuous-Integration-Infrastructure (phase-out-gallium): Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2366395 (10Dzahn) Same for labno... [23:29:28] 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 06Operations, and 2 others: Firewall rules for labs support host to communicate with contint1001.eqiad.wmnet (new gallium) - https://phabricator.wikimedia.org/T137323#2366543 (10Dzahn) |--|--|--|--|--|--|--|-- | TCP | scandium | 10.64.4.12 | contin... [23:35:13] 10Deployment-Systems, 03Scap3: Make puppet runs of deploy-local more configurable - https://phabricator.wikimedia.org/T131627#2366552 (10Ladsgroup) @mmodell The reason behind falls was actually an issue with checks that I couldn't solve at the moment. I can imagine a similar situation for others as well. It wo...