[00:18:22] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [00:18:22] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [00:49:23] 10Scap, 10ORES, 10Scoring-platform-team: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258 (10Ladsgroup) This hasn't happened to me in the past couple of months. Should I call it "automagically resolved"? [00:49:23] 10Scap, 10ORES, 10Scoring-platform-team: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258 (10Ladsgroup) This hasn't happened to me in the past couple of months. Should I call it "automagically resolved"? [00:54:39] 10Scap, 10ORES, 10Scoring-platform-team: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258 (10awight) >>! In T182258#4776389, @Ladsgroup wrote: > This hasn't happened to me in the past couple of months. Should I call it "automagically resolved"? Sure! Addition... [00:54:39] 10Scap, 10ORES, 10Scoring-platform-team: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258 (10awight) >>! In T182258#4776389, @Ladsgroup wrote: > This hasn't happened to me in the past couple of months. Should I call it "automagically resolved"? Sure! Addition... [00:54:44] 10Scap, 10ORES, 10Scoring-platform-team: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258 (10awight) 05Open>03Resolved a:03awight [00:54:44] 10Scap, 10ORES, 10Scoring-platform-team: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258 (10awight) 05Open>03Resolved a:03awight [05:26:32] 10Deployments, 10Operations, 10Operations-Software-Development: Make failures on foreachwiki more obvious the deployer - https://phabricator.wikimedia.org/T210474 (10dbarratt) [05:26:32] 10Deployments, 10Operations, 10Operations-Software-Development: Make failures on foreachwiki more obvious the deployer - https://phabricator.wikimedia.org/T210474 (10dbarratt) [05:26:48] 10Deployments, 10Operations, 10Operations-Software-Development: Make failures on foreachwiki more obvious the deployer - https://phabricator.wikimedia.org/T210474 (10dbarratt) [05:26:48] 10Deployments, 10Operations, 10Operations-Software-Development: Make failures on foreachwiki more obvious the deployer - https://phabricator.wikimedia.org/T210474 (10dbarratt) [06:17:24] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:17:24] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [06:27:23] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [06:27:23] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:26:50] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Wikimedia-Logstash, 10Patch-For-Review, 10Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324 (10hashar) One can find the logs by searching for `type:log4j`. log4j logs are emitted with `org.apache.log... [09:26:50] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Wikimedia-Logstash, 10Patch-For-Review, 10Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324 (10hashar) One can find the logs by searching for `type:log4j`. log4j logs are emitted with `org.apache.log... [09:29:29] 10Phabricator: Task dependency tree is unusable in mobile devices - https://phabricator.wikimedia.org/T210482 (10Agabi10) [09:29:29] 10Phabricator: Task dependency tree is unusable in mobile devices - https://phabricator.wikimedia.org/T210482 (10Agabi10) [09:47:28] 10Deployments, 10Operations: Make failures on foreachwiki more obvious the deployer - https://phabricator.wikimedia.org/T210474 (10Volans) [09:47:28] 10Deployments, 10Operations: Make failures on foreachwiki more obvious the deployer - https://phabricator.wikimedia.org/T210474 (10Volans) [10:12:21] 10Release-Engineering-Team (Kanban), 10docker-pkg, 10Patch-For-Review: docker-pkg should attempt to pull dependent images from the registry - https://phabricator.wikimedia.org/T200720 (10Joe) >>! In T200720#4468242, @hashar wrote: > I would rather enforce it. If a container `image:0.0.1` has been published... [10:12:21] 10Release-Engineering-Team (Kanban), 10docker-pkg, 10Patch-For-Review: docker-pkg should attempt to pull dependent images from the registry - https://phabricator.wikimedia.org/T200720 (10Joe) >>! In T200720#4468242, @hashar wrote: > I would rather enforce it. If a container `image:0.0.1` has been published... [10:22:04] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests: LDAP requests for Jeena Huneidi - https://phabricator.wikimedia.org/T210028 (10jcrespo) This is done, could @jeena try to use his LDAP credentials to log in to, for example, grafana to test everything is ok? [10:22:04] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests: LDAP requests for Jeena Huneidi - https://phabricator.wikimedia.org/T210028 (10jcrespo) This is done, could @jeena try to use his LDAP credentials to log in to, for example, grafana to test everything is ok? [11:15:15] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Zuul: Zuul merger should reject merge when Gerrit repository has Allow Content Merge set to false - https://phabricator.wikimedia.org/T210442 (10hashar) [11:15:15] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Zuul: Zuul merger should reject merge when Gerrit repository has Allow Content Merge set to false - https://phabricator.wikimedia.org/T210442 (10hashar) [11:38:41] !log Deploying ores:9b9ba06 to beta [11:38:41] !log Deploying ores:9b9ba06 to beta [11:38:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:38:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:30:20] Project beta-update-databases-eqiad build #30067: 04FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30067/ [12:30:20] Project beta-update-databases-eqiad build #30067: 04FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30067/ [12:41:51] hey greg-g can I have access to deployment-tin.deployment-prep ? [12:41:51] hey greg-g can I have access to deployment-tin.deployment-prep ? [12:42:40] I need to deploy service, I'm following the https://wikitech.wikimedia.org/wiki/Services/Deployment, but when I try to ssh to deploymnet-tin I get `channel 0: open failed: administratively prohibited: open failed` [12:42:41] I need to deploy service, I'm following the https://wikitech.wikimedia.org/wiki/Services/Deployment, but when I try to ssh to deploymnet-tin I get `channel 0: open failed: administratively prohibited: open failed` [12:42:58] I doint think that instance exists any more [12:42:58] I doint think that instance exists any more [12:43:15] It was rebuilt using a different name I think maybe deployment-deploy01 [12:43:15] It was rebuilt using a different name I think maybe deployment-deploy01 [12:44:52] paladox - thx - deployment-deploy01 worked [12:44:52] paladox - thx - deployment-deploy01 worked [12:45:28] Your welcome :) [12:45:28] Your welcome :) [13:00:00] 10Phabricator (Upstream), 10Mobile, 10Upstream: Task dependency tree is unusable in mobile devices - https://phabricator.wikimedia.org/T210482 (10Aklapper) p:05Triage>03Lowest That's why there is the search button instead in the upper right corner. [13:00:00] 10Phabricator (Upstream), 10Mobile, 10Upstream: Task dependency tree is unusable in mobile devices - https://phabricator.wikimedia.org/T210482 (10Aklapper) p:05Triage>03Lowest That's why there is the search button instead in the upper right corner. [13:11:38] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikimediaIncubator: Create a Wikimedia Incubator test wiki on the Beta Cluster - https://phabricator.wikimedia.org/T210492 (10Hydriz) [13:11:38] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-WikimediaIncubator: Create a Wikimedia Incubator test wiki on the Beta Cluster - https://phabricator.wikimedia.org/T210492 (10Hydriz) [13:15:00] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Zuul get confused when querying a change that yields two entries - https://phabricator.wikimedia.org/T198968 (10hashar) Upstream Zuul still have the issue: ` lang=python,name=zuul/driver/ge... [13:15:00] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Zuul get confused when querying a change that yields two entries - https://phabricator.wikimedia.org/T198968 (10hashar) Upstream Zuul still have the issue: ` lang=python,name=zuul/driver/ge... [13:21:28] Project beta-update-databases-eqiad build #30068: 04STILL FAILING in 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30068/ [13:21:28] Project beta-update-databases-eqiad build #30068: 04STILL FAILING in 36 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30068/ [13:56:47] oops [13:56:47] oops [13:57:01] I have forgot to cut the branch [13:57:01] I have forgot to cut the branch [13:59:03] !log cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/475352 [13:59:03] !log cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/475352 [13:59:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:59:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:05:34] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet fail on deployment-mediawiki-07, missing private hiera variable - https://phabricator.wikimedia.org/T210497 (10fgiunchedi) [14:05:34] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet fail on deployment-mediawiki-07, missing private hiera variable - https://phabricator.wikimedia.org/T210497 (10fgiunchedi) [14:16:21] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [14:16:21] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [14:21:29] Yippee, build fixed! [14:21:29] Yippee, build fixed! [14:21:29] Project beta-update-databases-eqiad build #30069: 09FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30069/ [14:21:29] Project beta-update-databases-eqiad build #30069: 09FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30069/ [14:26:21] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [14:26:21] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [14:28:24] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet fail on deployment-mediawiki-07, missing private hiera variable - https://phabricator.wikimedia.org/T210497 (10fgiunchedi) Temporarily unblocked/fixed by adding said variables to "project puppet" hiera config in Horizon [14:28:24] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet fail on deployment-mediawiki-07, missing private hiera variable - https://phabricator.wikimedia.org/T210497 (10fgiunchedi) Temporarily unblocked/fixed by adding said variables to "project puppet" hiera config in Horizon [14:31:06] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, and 2 others: Zuul get confused when querying a change that yields two entries - https://phabricator.wikimedia.org/T198968 (10hashar) Patch sent upstream: https://review.openstack.org/620328 gerrit:... [14:31:06] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, and 2 others: Zuul get confused when querying a change that yields two entries - https://phabricator.wikimedia.org/T198968 (10hashar) Patch sent upstream: https://review.openstack.org/620328 gerrit:... [14:37:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Upstream, 10Zuul: Zuul get confused when querying a change that yields two entries - https://phabricator.wikimedia.org/T198968 (10hashar) It is not a Gerrit issue but a bug in Zuul. [14:37:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Upstream, 10Zuul: Zuul get confused when querying a change that yields two entries - https://phabricator.wikimedia.org/T198968 (10hashar) It is not a Gerrit issue but a bug in Zuul. [14:44:34] 10Deployments, 10Operations: Make failures on foreachwiki more obvious the deployer - https://phabricator.wikimedia.org/T210474 (10Anomie) > Perhaps the failures could be gathered and displayed at the end? What's a "failure"? As mentioned at T209674#4754656, it seems unlikely that the generic `foreachwiki` s... [14:44:34] 10Deployments, 10Operations: Make failures on foreachwiki more obvious the deployer - https://phabricator.wikimedia.org/T210474 (10Anomie) > Perhaps the failures could be gathered and displayed at the end? What's a "failure"? As mentioned at T209674#4754656, it seems unlikely that the generic `foreachwiki` s... [15:55:17] !log giving User:Fisch-WMDE and User:Thiemo Mättig (WMDE) admin rights on enwiki, dewiki and arwiki (WMDE staff, beta cluster) [15:55:17] !log giving User:Fisch-WMDE and User:Thiemo Mättig (WMDE) admin rights on enwiki, dewiki and arwiki (WMDE staff, beta cluster) [15:55:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:55:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:50:08] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10Andrew) a:03Andrew [16:50:08] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10decommission, and 2 others: Remove labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T209642 (10Andrew) a:03Andrew [16:51:01] hashar: if you have time today could you please have another look at https://gerrit.wikimedia.org/r/c/integration/config/+/475496 ? [16:51:01] hashar: if you have time today could you please have another look at https://gerrit.wikimedia.org/r/c/integration/config/+/475496 ? [16:54:35] quibble question, I'm trying to reproduce a CI build but the PHPunit step fails with "PHPUnit not found. Please install it and other dev dependencies". If I run `composer install/update` within the container then a bunch of important dependencies are removed (like wikibase), so then I end up with errors like "Fatal error: Class undefined" [16:54:35] quibble question, I'm trying to reproduce a CI build but the PHPunit step fails with "PHPUnit not found. Please install it and other dev dependencies". If I run `composer install/update` within the container then a bunch of important dependencies are removed (like wikibase), so then I end up with errors like "Fatal error: Class undefined" [16:54:55] has anyone encountered this before? [16:54:55] has anyone encountered this before? [16:58:41] 10Continuous-Integration-Infrastructure (Slipway), 10Browser-Tests, 10User-zeljkofilipin: Migrate selenium-Wikibase-chrome selenium-WikibaseLexeme-chrome to Docker containers - https://phabricator.wikimedia.org/T210285 (10zeljkofilipin) I think `-chrome` is because they use a local browser (Chrome) while the... [16:58:41] 10Continuous-Integration-Infrastructure (Slipway), 10Browser-Tests, 10User-zeljkofilipin: Migrate selenium-Wikibase-chrome selenium-WikibaseLexeme-chrome to Docker containers - https://phabricator.wikimedia.org/T210285 (10zeljkofilipin) I think `-chrome` is because they use a local browser (Chrome) while the... [16:59:34] 10Continuous-Integration-Infrastructure (Slipway), 10Browser-Tests, 10User-zeljkofilipin: Migrate selenium-Wikibase-chrome selenium-WikibaseLexeme-chrome to Docker containers - https://phabricator.wikimedia.org/T210285 (10zeljkofilipin) [16:59:34] 10Continuous-Integration-Infrastructure (Slipway), 10Browser-Tests, 10User-zeljkofilipin: Migrate selenium-Wikibase-chrome selenium-WikibaseLexeme-chrome to Docker containers - https://phabricator.wikimedia.org/T210285 (10zeljkofilipin) [17:00:59] 10MediaWiki-Releasing, 10Phabricator, 10Project-Admins, 10Security-team-backlog: Create "security pre-announce" group in Phab (to easier allow 3rd parties who get pre-release notifications to access Security tasks) - https://phabricator.wikimedia.org/T108360 (10chasemp) 05stalled>03Open a:03chasemp [17:00:59] 10MediaWiki-Releasing, 10Phabricator, 10Project-Admins, 10Security-team-backlog: Create "security pre-announce" group in Phab (to easier allow 3rd parties who get pre-release notifications to access Security tasks) - https://phabricator.wikimedia.org/T108360 (10chasemp) 05stalled>03Open a:03chasemp [17:06:11] kostajh: sorry went busy [17:06:11] kostajh: sorry went busy [17:06:36] kostajh: that depends on the job / how you run it :) [17:06:36] kostajh: that depends on the job / how you run it :) [17:06:47] some job use dependencies from mediawiki/vendor [17:06:47] some job use dependencies from mediawiki/vendor [17:07:04] we then have some logic which look at mediawiki/core composer.json, extract the dev dependencies and the install them on top [17:07:04] we then have some logic which look at mediawiki/core composer.json, extract the dev dependencies and the install them on top [17:08:10] https://www.irccloud.com/pastebin/Y7twN0lP/ [17:08:10] https://www.irccloud.com/pastebin/Y7twN0lP/ [17:08:23] hashar ^ I think I copied everything over correctly for the `.env` file [17:08:23] hashar ^ I think I copied everything over correctly for the `.env` file [17:10:49] oh .env file [17:10:50] oh .env file [17:10:59] maybe we should have the CI job to attach it to the build [17:11:00] maybe we should have the CI job to attach it to the build [17:11:21] ZUUL_URL=git://contint2001.wikimedia.org [17:11:22] ZUUL_URL=git://contint2001.wikimedia.org [17:11:28] that one is not reacheable from outside/internet :( [17:11:29] that one is not reacheable from outside/internet :( [17:13:10] kostajh: can you paste some los from the build ? [17:13:10] kostajh: can you paste some los from the build ? [17:13:18] hashar: to back up a bit. What I'm trying to do is reproduce the inconsistent integration test failures noted in T208988. So maybe there is a simpler or faster way to do this besides reproducing the full build? [17:13:18] hashar: to back up a bit. What I'm trying to do is reproduce the inconsistent integration test failures noted in T208988. So maybe there is a simpler or faster way to do this besides reproducing the full build? [17:13:19] T208988: Fix Flow random test failures - https://phabricator.wikimedia.org/T208988 [17:13:19] T208988: Fix Flow random test failures - https://phabricator.wikimedia.org/T208988 [17:14:41] hashar: first I ran `docker run -it --rm --env-file ./.env -v "$(pwd)"/cache:/cache -v "$(pwd)"/log:/log -v "$(pwd)"/ref:/srv/git:ro -v "$(pwd)"/src:/workspace/src docker-registry.wikimedia.org/releng/quibble-jessie-hhvm` but after a long time that errored with `stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/p/mediawiki/extensions/NavigationTiming/': Failed to connect to [17:14:41] hashar: first I ran `docker run -it --rm --env-file ./.env -v "$(pwd)"/cache:/cache -v "$(pwd)"/log:/log -v "$(pwd)"/ref:/srv/git:ro -v "$(pwd)"/src:/workspace/src docker-registry.wikimedia.org/releng/quibble-jessie-hhvm` but after a long time that errored with `stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/p/mediawiki/extensions/NavigationTiming/': Failed to connect to [17:14:41] gerrit.wikimedia.org port 443: Connection refused'`. [17:14:42] gerrit.wikimedia.org port 443: Connection refused'`. [17:15:18] then I ran `docker run -it --rm --env-file ./.env -v "$(pwd)"/cache:/cache -v "$(pwd)"/log:/log -v "$(pwd)"/ref:/srv/git:ro -v "$(pwd)"/src:/workspace/src docker-registry.wikimedia.org/releng/quibble-jessie-hhvm --skip-zuul --skip-deps --commands "php tests/phpunit/phpunit.php extensions/Flow/tests/phpunit --filter=testGetLastRevision"` but that errored with "PHPUnit not found". [17:15:18] then I ran `docker run -it --rm --env-file ./.env -v "$(pwd)"/cache:/cache -v "$(pwd)"/log:/log -v "$(pwd)"/ref:/srv/git:ro -v "$(pwd)"/src:/workspace/src docker-registry.wikimedia.org/releng/quibble-jessie-hhvm --skip-zuul --skip-deps --commands "php tests/phpunit/phpunit.php extensions/Flow/tests/phpunit --filter=testGetLastRevision"` but that errored with "PHPUnit not found". [17:15:42] so maybe I need to re-run the first command and not add `--skip-zuul` or `--skip-deps` to ensure that everything gets installed properly. [17:15:42] so maybe I need to re-run the first command and not add `--skip-zuul` or `--skip-deps` to ensure that everything gets installed properly. [17:15:45] hmm [17:15:45] hmm [17:15:58] for me anyway this seems to take like 20-30 minutes, is that your experience as well? [17:15:58] for me anyway this seems to take like 20-30 minutes, is that your experience as well? [17:16:15] so --skip-zuul is to prevent quibble from fetching/resetting the git repos [17:16:15] so --skip-zuul is to prevent quibble from fetching/resetting the git repos [17:16:27] --skip-deps is when you know you have kept the result of composer install / npm install [17:16:27] --skip-deps is when you know you have kept the result of composer install / npm install [17:16:38] so usually yeah one can keep them around [17:16:38] so usually yeah one can keep them around [17:17:10] seems git cant reach gerrit somehow :( [17:17:10] seems git cant reach gerrit somehow :( [17:17:17] maybe the docker container does not have network access [17:17:17] maybe the docker container does not have network access [17:17:52] might want to try: docker run -it --rm --entrypoint=git docker-registry.wikimedia.org/releng/quibble-jessie-hhvm ls-remote https://gerrit.wikimedia.org/r/p/mediawiki/extensions/NavigationTiming [17:17:52] might want to try: docker run -it --rm --entrypoint=git docker-registry.wikimedia.org/releng/quibble-jessie-hhvm ls-remote https://gerrit.wikimedia.org/r/p/mediawiki/extensions/NavigationTiming [17:18:04] (which runs git ls-remote in a container) [17:18:04] (which runs git ls-remote in a container) [17:18:21] for phpunit not found hmm [17:18:21] for phpunit not found hmm [17:18:52] php tests/phpunit/phpunit.php should find it [17:18:52] php tests/phpunit/phpunit.php should find it [17:19:59] kostajh: so I don't know :(((( [17:19:59] kostajh: so I don't know :(((( [17:20:14] kostajh: also you can run quibble directly on your machine without a container, might be easier [17:20:14] kostajh: also you can run quibble directly on your machine without a container, might be easier [17:20:31] hashar: it's cloning repos fine, but sometimes it seems to give up [17:20:31] hashar: it's cloning repos fine, but sometimes it seems to give up [17:21:20] :( [17:21:20] :( [17:21:27] on a related note I filed T210435 [17:21:27] on a related note I filed T210435 [17:21:28] T210435: Use git clone depth=1 with zuul? - https://phabricator.wikimedia.org/T210435 [17:21:28] T210435: Use git clone depth=1 with zuul? - https://phabricator.wikimedia.org/T210435 [17:21:39] oh [17:21:40] oh [17:21:43] but it looks like you made note of this many years ago in T70481 [17:21:43] but it looks like you made note of this many years ago in T70481 [17:21:44] T70481: Zuul repositories have too many refs causing slow updates - https://phabricator.wikimedia.org/T70481 [17:21:44] T70481: Zuul repositories have too many refs causing slow updates - https://phabricator.wikimedia.org/T70481 [17:21:55] :) [17:21:55] :) [17:22:14] that T70481 is for the zuul infra which indeed keep crafting and pilling up ref over and over [17:22:14] that T70481 is for the zuul infra which indeed keep crafting and pilling up ref over and over [17:22:22] hashar: I know i could run it without a container but I suspect the issues might be due to the container environment, so that's why I want to try to reproduce there [17:22:22] hashar: I know i could run it without a container but I suspect the issues might be due to the container environment, so that's why I want to try to reproduce there [17:22:23] that causes CI git operations to slowdown slightly over time [17:22:23] that causes CI git operations to slowdown slightly over time [17:22:37] for depth=1 , in practice it does not work on CI for various reason [17:22:37] for depth=1 , in practice it does not work on CI for various reason [17:22:59] but at least we could do the clone and fetch in parallel instead of serially [17:22:59] but at least we could do the clone and fetch in parallel instead of serially [17:23:22] also I think you can let quibble run and just drop in the shell for debugging [17:23:22] also I think you can let quibble run and just drop in the shell for debugging [17:24:07] probably docker run -it --rm docker-registry.wikimedia.org/releng/quibble-jessie-hhvm --skip-zuul --skip-deps --commands=/bin/bash [17:24:07] probably docker run -it --rm docker-registry.wikimedia.org/releng/quibble-jessie-hhvm --skip-zuul --skip-deps --commands=/bin/bash [17:24:22] which should install mediawiki, boot db, start chromedriver and webserver [17:24:22] which should install mediawiki, boot db, start chromedriver and webserver [17:24:27] then give you a shell prompt to play with [17:24:27] then give you a shell prompt to play with [17:24:37] eg running phpunit / debugging why it fails [17:24:37] eg running phpunit / debugging why it fails [17:25:11] given the source is mounted in the container form the host, you should be able to edit files with your usual editor [17:25:11] given the source is mounted in the container form the host, you should be able to edit files with your usual editor [17:25:18] kostajh: sorry spurting ideas out loud :( [17:25:19] kostajh: sorry spurting ideas out loud :( [17:26:29] hashar: I have tried that, however phpunit isn't installed, and if I do `composer update` it will remove a number of repos, like `wikibase`, so then I get "class not found" errors. [17:26:29] hashar: I have tried that, however phpunit isn't installed, and if I do `composer update` it will remove a number of repos, like `wikibase`, so then I get "class not found" errors. [17:26:55] Maybe there is another command besides `composer update` that will fetch core's composer requirements in addition to whatever is in the root composer.json for that build? [17:26:55] Maybe there is another command besides `composer update` that will fetch core's composer requirements in addition to whatever is in the root composer.json for that build? [17:28:29] kostajh: I think the default is to use mediawiki/vendor.git [17:28:29] kostajh: I think the default is to use mediawiki/vendor.git [17:28:35] then [17:28:35] then [17:28:48] at the root of the mediawiki/core install, Quibble should have created a file composer.local.json [17:28:48] at the root of the mediawiki/core install, Quibble should have created a file composer.local.json [17:28:56] which list a bunch of composer.json from extensions and skins [17:28:56] which list a bunch of composer.json from extensions and skins [17:29:14] running compoesr install should trigger the composer-merge-plugin which read composer.local.json and then all referenced composer.json files [17:29:14] running compoesr install should trigger the composer-merge-plugin which read composer.local.json and then all referenced composer.json files [17:29:33] it then rebuild the list of dependencies and install whatever is missing into ./vendor/ (which is a clone of mediawiki/vendor.git ) [17:29:33] it then rebuild the list of dependencies and install whatever is missing into ./vendor/ (which is a clone of mediawiki/vendor.git ) [17:30:37] hmm. Ok I'll see what happens with this build. So far I just see composer.json, not composer.local.json. [17:30:37] hmm. Ok I'll see what happens with this build. So far I just see composer.json, not composer.local.json. [17:49:34] kostajh: good luck, and copy past the whole output somewhere and attach to the task :) [17:49:42] will see wha tI can do out of that or whether I reproduce [17:49:48] good luck. I am rushing back home [18:18:19] hi jdlrobson [18:18:19] hi jdlrobson [18:18:22] hey Krenair [18:18:22] hey Krenair [18:18:36] https://www.irccloud.com/pastebin/tb4CoLGr/ [18:18:36] https://www.irccloud.com/pastebin/tb4CoLGr/ [18:18:37] when selenium tries to edit pages [18:18:37] when selenium tries to edit pages [18:18:37] i'm seeing selenium failures across the board but can't replicate [18:18:37] i'm seeing selenium failures across the board but can't replicate [18:18:37] okay but what's the actual HTTP request it's making here? [18:18:37] okay but what's the actual HTTP request it's making here? [18:18:40] jdlrobson, we could have a naive attempt at triggering it ourselves by figuring out what wiki and page it's trying to edit and doing something similar [18:18:40] jdlrobson, we could have a naive attempt at triggering it ourselves by figuring out what wiki and page it's trying to edit and doing something similar [18:18:40] but really to debug this properly we should have the full request details :/ [18:18:40] but really to debug this properly we should have the full request details :/ [18:20:05] Krenair: so I can't replicate this with the API. So are you telling me that the wiki should not be read only at this point and that I can expect some stability? [18:20:05] Krenair: so I can't replicate this with the API. So are you telling me that the wiki should not be read only at this point and that I can expect some stability? [18:20:20] If so I can dig into this some more [18:20:20] If so I can dig into this some more [18:20:30] yes [18:20:30] yes [18:20:45] as far as I know the beta wiki is stable [18:20:45] as far as I know the beta wiki is stable [18:21:08] the weird thing is the `And the page "Selenium search test" exists` step is generic and tests using it are passing. So it seems to depend on the title (?!) [18:21:08] the weird thing is the `And the page "Selenium search test" exists` step is generic and tests using it are passing. So it seems to depend on the title (?!) [18:21:14] if you can give me a curl command to run against its API that generates a read only error I will look into it [18:21:14] if you can give me a curl command to run against its API that generates a read only error I will look into it [18:28:57] 10Release-Engineering-Team (Kanban), 10User-greg: Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jcrespo) [18:28:57] 10Release-Engineering-Team (Kanban), 10User-greg: Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jcrespo) [18:29:28] 10Release-Engineering-Team (Kanban), 10User-greg: Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jcrespo) [18:29:28] 10Release-Engineering-Team (Kanban), 10User-greg: Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jcrespo) [18:29:30] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests: LDAP requests for Jeena Huneidi - https://phabricator.wikimedia.org/T210028 (10jcrespo) 05Open>03Resolved Access request was tested, it was possible to loging to a bastion host, deployment host and to grafana, so this is done! [18:29:30] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests: LDAP requests for Jeena Huneidi - https://phabricator.wikimedia.org/T210028 (10jcrespo) 05Open>03Resolved Access request was tested, it was possible to loging to a bastion host, deployment host and to grafana, so this is done! [18:38:49] It's odd... it's doing a simple POST to action edit e.g. https://en.wikipedia.beta.wmflabs.org/wiki/Special:ApiSandbox#action=edit&format=json&title=Selenium%20search%20test&text=Test%20is%20used%20by%20Selenium%20web%20driver&token=a58125cfef2cf4b200e4953376ace77d5bfd8ba0%2B%5C but the edit API is responding with the read only flag. dieReadOnly is called when EditPage::AS_READ_ONLY_PAGE and ApiBase::execute is a [18:38:49] It's odd... it's doing a simple POST to action edit e.g. https://en.wikipedia.beta.wmflabs.org/wiki/Special:ApiSandbox#action=edit&format=json&title=Selenium%20search%20test&text=Test%20is%20used%20by%20Selenium%20web%20driver&token=a58125cfef2cf4b200e4953376ace77d5bfd8ba0%2B%5C but the edit API is responding with the read only flag. dieReadOnly is called when EditPage::AS_READ_ONLY_PAGE and ApiBase::execute is a [18:38:49] huggeee function. It's hard to work out why that would be the case. [18:38:49] huggeee function. It's hard to work out why that would be the case. [18:41:53] I have a theory im now exploring.. [18:41:53] I have a theory im now exploring.. [18:42:13] 10Release-Engineering-Team (Watching / External), 10Phlogiston (Setting Up Reports), 10User-greg: Adjust phlogiston configuration for Release Engineering - https://phabricator.wikimedia.org/T170359 (10greg) 05Open>03Resolved Closing out the old task :) Thanks for your help here in the past Joel. [18:42:13] 10Release-Engineering-Team (Watching / External), 10Phlogiston (Setting Up Reports), 10User-greg: Adjust phlogiston configuration for Release Engineering - https://phabricator.wikimedia.org/T170359 (10greg) 05Open>03Resolved Closing out the old task :) Thanks for your help here in the past Joel. [18:49:05] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Random test failure Flow\Tests\Api\ApiFlowModerateTopicTest::testModerateLockedTopic and Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T209729 (10Daim... [18:49:05] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Random test failure Flow\Tests\Api\ApiFlowModerateTopicTest::testModerateLockedTopic and Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T209729 (10Daim... [18:50:30] 10Release-Engineering-Team, 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.9-1 - https://phabricator.wikimedia.org/T210469 (10thcipriani) I checked the command run by sync-wikiversions after the package was updated, and that looked correct; however, I now realize that the s... [18:50:30] 10Release-Engineering-Team, 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.9-1 - https://phabricator.wikimedia.org/T210469 (10thcipriani) I checked the command run by sync-wikiversions after the package was updated, and that looked correct; however, I now realize that the s... [18:50:35] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), and 2 others: Flaky unit test Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T210204 (10Daimona) [18:50:35] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), and 2 others: Flaky unit test Flow\Tests\Collection\PostCollectionTest::testGetLastRevision - https://phabricator.wikimedia.org/T210204 (10Daimona) [18:51:51] 10Release-Engineering-Team, 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.10-1 - https://phabricator.wikimedia.org/T210469 (10thcipriani) [18:51:51] 10Release-Engineering-Team, 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.10-1 - https://phabricator.wikimedia.org/T210469 (10thcipriani) [19:01:08] 10Gerrit, 10Release-Engineering-Team (Kanban), 10User-greg: Enable 2FA on wmfgerrit github account - https://phabricator.wikimedia.org/T198822 (10greg) p:05Triage>03Normal a:03greg I'll do this sometime this week and add the relevant info to our pwstore. [19:01:08] 10Gerrit, 10Release-Engineering-Team (Kanban), 10User-greg: Enable 2FA on wmfgerrit github account - https://phabricator.wikimedia.org/T198822 (10greg) p:05Triage>03Normal a:03greg I'll do this sometime this week and add the relevant info to our pwstore. [19:05:26] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Scap: Refresh Jenkins API token for Harbormaster - https://phabricator.wikimedia.org/T207335 (10greg) p:05Triage>03Normal [19:05:26] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Scap: Refresh Jenkins API token for Harbormaster - https://phabricator.wikimedia.org/T207335 (10greg) p:05Triage>03Normal [19:10:16] 10Release-Engineering-Team, 10Scap, 10Operations: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10greg) random question from from this task and T203664: should the debug hosts also be the same hardware/setup as the main clu... [19:10:16] 10Release-Engineering-Team, 10Scap, 10Operations: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10greg) random question from from this task and T203664: should the debug hosts also be the same hardware/setup as the main clu... [19:11:31] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Zuul: Make Jenkins jobs fail if part of setup-zuul-submodules fails - https://phabricator.wikimedia.org/T199077 (10greg) [19:11:31] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Zuul: Make Jenkins jobs fail if part of setup-zuul-submodules fails - https://phabricator.wikimedia.org/T199077 (10greg) [19:12:23] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team: search-mjolnir-tox-docker job in status ABORT somewhat regularly - https://phabricator.wikimedia.org/T190963 (10greg) >>! In T190963#4404212, @greg wrote: > And because our jenkins history is time limited I can now only see the las... [19:12:23] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team: search-mjolnir-tox-docker job in status ABORT somewhat regularly - https://phabricator.wikimedia.org/T190963 (10greg) >>! In T190963#4404212, @greg wrote: > And because our jenkins history is time limited I can now only see the las... [19:13:12] 10Deployments, 10Release-Engineering-Team, 10TechCom-RFC, 10Core Platform Team ( Code Health (TEC13)), and 2 others: RFC: Reevaluate LocalisationUpdate extension for WMF - https://phabricator.wikimedia.org/T158360 (10greg) p:05Triage>03Normal [19:13:12] 10Deployments, 10Release-Engineering-Team, 10TechCom-RFC, 10Core Platform Team ( Code Health (TEC13)), and 2 others: RFC: Reevaluate LocalisationUpdate extension for WMF - https://phabricator.wikimedia.org/T158360 (10greg) p:05Triage>03Normal [19:14:29] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.10-1 - https://phabricator.wikimedia.org/T210469 (10greg) [19:14:29] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.10-1 - https://phabricator.wikimedia.org/T210469 (10greg) [19:15:17] 10Release-Engineering-Team (Backlog), 10Release Pipeline: Initial production image build fails helm test - https://phabricator.wikimedia.org/T209871 (10greg) [19:15:17] 10Release-Engineering-Team (Backlog), 10Release Pipeline: Initial production image build fails helm test - https://phabricator.wikimedia.org/T209871 (10greg) [19:15:38] 10Release-Engineering-Team (Backlog), 10Operations, 10Release Pipeline: Design pipeline image versioning scheme - https://phabricator.wikimedia.org/T209088 (10greg) [19:15:38] 10Release-Engineering-Team (Backlog), 10Operations, 10Release Pipeline: Design pipeline image versioning scheme - https://phabricator.wikimedia.org/T209088 (10greg) [19:19:23] 10Release-Engineering-Team, 10User-zeljkofilipin, 10Wikimedia-Incident, 10Wikimedia-production-error: Investigate the spikes of "web request took longer than 60 seconds and timed out" during deployments - https://phabricator.wikimedia.org/T204871 (10greg) [19:19:23] 10Release-Engineering-Team, 10User-zeljkofilipin, 10Wikimedia-Incident, 10Wikimedia-production-error: Investigate the spikes of "web request took longer than 60 seconds and timed out" during deployments - https://phabricator.wikimedia.org/T204871 (10greg) [19:19:25] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team: search-mjolnir-tox-docker job in status ABORT somewhat regularly - https://phabricator.wikimedia.org/T190963 (10EBernhardson) 05Open>03Resolved a:03EBernhardson seems reasonable lately. Lets call it good. [19:19:25] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team: search-mjolnir-tox-docker job in status ABORT somewhat regularly - https://phabricator.wikimedia.org/T190963 (10EBernhardson) 05Open>03Resolved a:03EBernhardson seems reasonable lately. Lets call it good. [19:20:12] 10Release-Engineering-Team, 10User-zeljkofilipin, 10Wikimedia-Incident, 10Wikimedia-production-error: Investigate the spikes of "web request took longer than 60 seconds and timed out" during deployments - https://phabricator.wikimedia.org/T204871 (10greg) Making this a follow-up/actionable to track this wo... [19:20:12] 10Release-Engineering-Team, 10User-zeljkofilipin, 10Wikimedia-Incident, 10Wikimedia-production-error: Investigate the spikes of "web request took longer than 60 seconds and timed out" during deployments - https://phabricator.wikimedia.org/T204871 (10greg) Making this a follow-up/actionable to track this wo... [19:22:26] 10Release-Engineering-Team (Backlog), 10Code-Health, 10Wikimedia-Incident: Look into ways to identify risky commits that are part of the weekly branch cut, either manually or through automated process - https://phabricator.wikimedia.org/T193258 (10Jrbranaa) [19:22:26] 10Release-Engineering-Team (Backlog), 10Code-Health, 10Wikimedia-Incident: Look into ways to identify risky commits that are part of the weekly branch cut, either manually or through automated process - https://phabricator.wikimedia.org/T193258 (10Jrbranaa) [19:23:13] 10Release-Engineering-Team, 10Scap, 10Operations: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10Legoktm) Yeah, that seems sensible unless there's some significant reason (e.g. hardware cost) not to. [19:23:13] 10Release-Engineering-Team, 10Scap, 10Operations: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10Legoktm) Yeah, that seems sensible unless there's some significant reason (e.g. hardware cost) not to. [19:28:23] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), 10Patch-For-Review: Fix Flow random test failures - https://phabricator.wikimedia.org/T208988 (10kostajh) @hashar `docker run -it --rm --env-file ./.env -v "$(pwd)"/cache:/cache -v "$(pwd)"/log... [19:28:23] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), 10Patch-For-Review: Fix Flow random test failures - https://phabricator.wikimedia.org/T208988 (10kostajh) @hashar `docker run -it --rm --env-file ./.env -v "$(pwd)"/cache:/cache -v "$(pwd)"/log... [19:28:35] nope.. i don't get this read only bug. How can API requests pass in one place but not in another!? [19:28:35] nope.. i don't get this read only bug. How can API requests pass in one place but not in another!? [19:29:06] 10Release-Engineering-Team, 10Code-Health: TEC13:O4.1:Q1 Goal - Define base Code Health metric set. - https://phabricator.wikimedia.org/T199261 (10Jrbranaa) Progress to date: - WG has deployed an evaluation of SonarQube as a tool for CH Metrics analysis and collection for the Search Platform team's Java code... [19:29:06] 10Release-Engineering-Team, 10Code-Health: TEC13:O4.1:Q1 Goal - Define base Code Health metric set. - https://phabricator.wikimedia.org/T199261 (10Jrbranaa) Progress to date: - WG has deployed an evaluation of SonarQube as a tool for CH Metrics analysis and collection for the Search Platform team's Java code... [19:31:06] hi thcipriani greg-g .. i had a qn .. so, we discovered a pretty bad regression in parsoid caused by a parsoid-batch-api extension change .. that went out Nov 14 .. we just reverted the change. [19:31:06] hi thcipriani greg-g .. i had a qn .. so, we discovered a pretty bad regression in parsoid caused by a parsoid-batch-api extension change .. that went out Nov 14 .. we just reverted the change. [19:31:09] 10Release-Engineering-Team: TEC13:O3.2:Q1 Goal - Platform and Search Platform teams are using Technical Debt Management PoC - https://phabricator.wikimedia.org/T199259 (10Jrbranaa) Progress to date: - Slow going at first as this is a shared goal. Originally a Q1 goal, this has moved to a Q2 goal in order to a... [19:31:09] 10Release-Engineering-Team: TEC13:O3.2:Q1 Goal - Platform and Search Platform teams are using Technical Debt Management PoC - https://phabricator.wikimedia.org/T199259 (10Jrbranaa) Progress to date: - Slow going at first as this is a shared goal. Originally a Q1 goal, this has moved to a Q2 goal in order to a... [19:31:11] i know the branch is probablycut now. [19:31:11] i know the branch is probablycut now. [19:31:27] wondering if there is any way to get that onto the train before it goes out .. or should we try to do it with swat? [19:31:27] wondering if there is any way to get that onto the train before it goes out .. or should we try to do it with swat? [19:31:46] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ParsoidBatchAPI/+/476088 is the change in qn. [19:31:46] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ParsoidBatchAPI/+/476088 is the change in qn. [19:31:50] subbu: do a backport+swat [19:31:51] subbu: do a backport+swat [19:32:14] or backport+deploynow! if it's that important :) [19:32:14] or backport+deploynow! if it's that important :) [19:32:14] ok. [19:32:14] ok. [19:32:19] we're just on group0 now [19:32:19] we're just on group0 now [19:32:48] 10Release-Engineering-Team: TEC13:O2.3:Q1 Goal - make current unit testing coverage more visible by reporting out to Engineering Management. - https://phabricator.wikimedia.org/T199257 (10Jrbranaa) 05Open>03Resolved a:03Jrbranaa Current unit test coverage has become part of the Code Health newsletter. Cod... [19:32:48] 10Release-Engineering-Team: TEC13:O2.3:Q1 Goal - make current unit testing coverage more visible by reporting out to Engineering Management. - https://phabricator.wikimedia.org/T199257 (10Jrbranaa) 05Open>03Resolved a:03Jrbranaa Current unit test coverage has become part of the Code Health newsletter. Cod... [19:33:38] 10Release-Engineering-Team: TEC13:O2.1:Q1 Goal - Add test evaluation to post mortem review process. - https://phabricator.wikimedia.org/T199254 (10Jrbranaa) 05Open>03Resolved a:03Jrbranaa The post mortem process now includes a section on test/escape analysis. [19:33:38] 10Release-Engineering-Team: TEC13:O2.1:Q1 Goal - Add test evaluation to post mortem review process. - https://phabricator.wikimedia.org/T199254 (10Jrbranaa) 05Open>03Resolved a:03Jrbranaa The post mortem process now includes a section on test/escape analysis. [19:47:28] 10Release-Engineering-Team (Kanban), 10Scap: deploy1001:/srv/mediawiki out of date? - https://phabricator.wikimedia.org/T207602 (10thcipriani) [19:47:28] 10Release-Engineering-Team (Kanban), 10Scap: deploy1001:/srv/mediawiki out of date? - https://phabricator.wikimedia.org/T207602 (10thcipriani) [19:47:35] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.10-1 - https://phabricator.wikimedia.org/T210469 (10thcipriani) 05Open>03Resolved a:05thcipriani>03colewhite thanks @fgiunchedi for the initial 3.8.9-1 release (sorry for missing ini... [19:47:35] 10Release-Engineering-Team (Kanban), 10Scap, 10Operations, 10Patch-For-Review: Update Debian Package for Scap to 3.8.10-1 - https://phabricator.wikimedia.org/T210469 (10thcipriani) 05Open>03Resolved a:05thcipriani>03colewhite thanks @fgiunchedi for the initial 3.8.9-1 release (sorry for missing ini... [19:48:01] 10Release-Engineering-Team (Kanban), 10Scap: deploy1001:/srv/mediawiki out of date? - https://phabricator.wikimedia.org/T207602 (10thcipriani) 05Open>03Resolved Now that scap 3.8.10-1 is out, this should be fixed! [19:48:01] 10Release-Engineering-Team (Kanban), 10Scap: deploy1001:/srv/mediawiki out of date? - https://phabricator.wikimedia.org/T207602 (10thcipriani) 05Open>03Resolved Now that scap 3.8.10-1 is out, this should be fixed! [19:59:40] Who maintains the mediawiki_api ruby gem? [19:59:40] Who maintains the mediawiki_api ruby gem? [20:15:05] at least 10 times i have clicked =the Code of Conduct footer in Gerrit without meaning to. each time i am in Edit mode and want to hit Save or Exit.. there is the footer link in the way.. arrr [20:15:05] at least 10 times i have clicked =the Code of Conduct footer in Gerrit without meaning to. each time i am in Edit mode and want to hit Save or Exit.. there is the footer link in the way.. arrr [20:15:49] mutante: me too!!! [20:15:49] mutante: me too!!! [20:15:54] im glad i'm not alone! when editing commit msg [20:15:54] im glad i'm not alone! when editing commit msg [20:16:00] yea,, only in GWTUI, but that is still default [20:16:00] yea,, only in GWTUI, but that is still default [20:16:07] yes:) [20:16:07] yes:) [20:34:59] the GWTUI has been removed upstream (the bug should be fixed in PolyGerrit!) [20:34:59] the GWTUI has been removed upstream (the bug should be fixed in PolyGerrit!) [20:36:00] yea, but gwtui is still default in gerrit and polygerrit is opt-in [20:36:00] yea, but gwtui is still default in gerrit and polygerrit is opt-in [20:36:31] for wikimedia gerrit that is [20:36:31] for wikimedia gerrit that is [20:36:36] yeh, but the bug won't be fixed unless (someone else fixes it) [20:36:36] yeh, but the bug won't be fixed unless (someone else fixes it) [20:36:46] or it's a critical fix [20:36:46] or it's a critical fix [20:37:39] what bug exactly though? that the footer content is in that place? [20:37:39] what bug exactly though? that the footer content is in that place? [20:38:02] the fact the footer is not correctly placed (this is a bug internally in gerrit). [20:38:02] the fact the footer is not correctly placed (this is a bug internally in gerrit). [20:38:09] ok, gotcha [20:38:09] ok, gotcha [20:38:48] can it be influenced by changing some Edit Preferences maybe? [20:38:48] can it be influenced by changing some Edit Preferences maybe? [20:38:58] Nope, i doin't think so [20:38:58] Nope, i doin't think so [20:39:02] hm,ok [20:39:02] hm,ok [20:45:31] 10Project-Admins: Create a component project for the EditAccount extension - https://phabricator.wikimedia.org/T205293 (10ashley) EditAccount doesn't currently support AuthManager (T135264) and is bascially abandonware at this point. Given that and the fact that [[https://www.mediawiki.org/wiki/Extension:EditUse... [20:45:31] 10Project-Admins: Create a component project for the EditAccount extension - https://phabricator.wikimedia.org/T205293 (10ashley) EditAccount doesn't currently support AuthManager (T135264) and is bascially abandonware at this point. Given that and the fact that [[https://www.mediawiki.org/wiki/Extension:EditUse... [20:59:04] well how about that, that is a terrible placement in that screen. [20:59:04] well how about that, that is a terrible placement in that screen. [21:02:16] 10Release-Engineering-Team, 10Scap, 10Operations: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10greg) >>! In T203625#4778902, @Legoktm wrote: >>>! In T203625#4778825, @greg wrote: >> random question from from this task an... [21:02:16] 10Release-Engineering-Team, 10Scap, 10Operations: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10greg) >>! In T203625#4778902, @Legoktm wrote: >>>! In T203625#4778825, @greg wrote: >> random question from from this task an... [21:08:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team, 10Patch-For-Review: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10greg) p:05Triage>03Normal Are we good here? [21:08:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team, 10Patch-For-Review: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10greg) p:05Triage>03Normal Are we good here? [21:09:57] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10greg) a:03hashar [21:09:57] 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate the Integration cloud project to eqiad1-r - https://phabricator.wikimedia.org/T208803 (10greg) a:03hashar [21:12:10] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10greg) a:03Krenair While @mmodell was the point person from #release-engineering-team, @Krenair did much of t... [21:12:10] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10greg) a:03Krenair While @mmodell was the point person from #release-engineering-team, @Krenair did much of t... [21:13:29] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10greg) [21:13:29] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Q1 Selenium framework improvements - https://phabricator.wikimedia.org/T198389 (10greg) [21:13:31] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10greg) 05Open>03Resolved | Repository | Change | Maintainers | Coverage | Incidents | Selected | |--|--|--|--|--|--| | medi... [21:13:31] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10greg) 05Open>03Resolved | Repository | Change | Maintainers | Coverage | Incidents | Selected | |--|--|--|--|--|--| | medi... [21:14:01] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10greg) p:05Triage>03Normal [21:14:01] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10greg) p:05Triage>03Normal [21:23:29] thcipriani: thanks! merged and applied CSS change. though, not sure if can confirm it's gone yet [21:23:29] thcipriani: thanks! merged and applied CSS change. though, not sure if can confirm it's gone yet [21:25:51] thcipriani i found a fix to ti [21:25:51] thcipriani i found a fix to ti [21:25:53] *it [21:25:53] *it [21:26:02] removing "position: absolute" fixes t [21:26:02] removing "position: absolute" fixes t [21:26:03] mutante: ah, looks like the css I was trying to override gets loaded a bit later, looks like I need to use a stronger selector [21:26:03] mutante: ah, looks like the css I was trying to override gets loaded a bit later, looks like I need to use a stronger selector [21:26:05] *it [21:26:05] *it [21:28:54] thcipriani mutante [21:28:54] thcipriani mutante [21:28:55] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/476142/ [21:28:55] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/476142/ [21:30:13] paladox: that messes with the "powered by" line though [21:30:13] paladox: that messes with the "powered by" line though [21:30:17] makes it appear under it [21:30:17] makes it appear under it [21:30:22] yep [21:30:22] yep [21:30:24] hmm [21:30:24] hmm [21:30:48] I think I've got it [21:30:48] I think I've got it [21:33:12] https://gerrit.wikimedia.org/r/c/operations/puppet/+/476144 [21:33:12] https://gerrit.wikimedia.org/r/c/operations/puppet/+/476144 [21:33:32] I should have hired front end engineers :P [21:33:32] I should have hired front end engineers :P [21:33:49] lol [21:33:49] lol [21:34:07] thcipriani it will be much easier to modify polygerrit :) [21:34:07] thcipriani it will be much easier to modify polygerrit :) [21:37:18] greg-g: too expensive, you're stuck with me instead [21:37:18] greg-g: too expensive, you're stuck with me instead [21:37:19] i tested that by manually copyign the line [21:37:19] i tested that by manually copyign the line [21:37:38] thcipriani: bummer [21:37:38] thcipriani: bummer [21:37:43] :P [21:37:43] :P [21:40:05] 10Project-Admins: Create a component project for the EditAccount extension - https://phabricator.wikimedia.org/T205293 (10Aklapper) @ashley: Thanks for chiming in! Should this be reflected on https://www.mediawiki.org/wiki/Extension:EditAccount and its 'status' be changed? [21:40:05] 10Project-Admins: Create a component project for the EditAccount extension - https://phabricator.wikimedia.org/T205293 (10Aklapper) @ashley: Thanks for chiming in! Should this be reflected on https://www.mediawiki.org/wiki/Extension:EditAccount and its 'status' be changed? [21:40:52] i deployed it, puppet was noop because i had already manually pasted the line [21:40:52] i deployed it, puppet was noop because i had already manually pasted the line [21:41:20] paladox could confirm it, i could not [21:41:20] paladox could confirm it, i could not [21:43:35] strange, works for me now as well: http://tyler.zone/no-more-gerrit-footer.png [21:43:35] strange, works for me now as well: http://tyler.zone/no-more-gerrit-footer.png [21:43:37] thcipriani out of curiosity do you know where that Edit class is? I carn't seem to find the class in gerrit. [21:43:37] thcipriani out of curiosity do you know where that Edit class is? I carn't seem to find the class in gerrit. [21:44:28] my bad, confirmed after fully restarting my browser [21:44:28] my bad, confirmed after fully restarting my browser [21:44:31] thanks for fixing :)) [21:44:31] thanks for fixing :)) [21:44:38] jdlrobson: ^ :) [21:44:38] jdlrobson: ^ :) [21:44:59] * paladox edits with the new inline editor https://phabricator.wikimedia.org/F27319661 [21:44:59] * paladox edits with the new inline editor https://phabricator.wikimedia.org/F27319661 [21:46:35] paladox: not sure, gerrit-gwtui/src/main/java/com/google/gerrit/client/editor/EditScreen.java seems related anyway [21:46:35] paladox: not sure, gerrit-gwtui/src/main/java/com/google/gerrit/client/editor/EditScreen.java seems related anyway [21:46:57] thcipriani ohhh [21:46:57] thcipriani ohhh [21:46:58] i know why [21:46:58] i know why [21:47:13] they removed gwtui source code from the master branch so github searching returns nothing [21:47:13] they removed gwtui source code from the master branch so github searching returns nothing [21:47:36] ah, yep, that'd do it :) [21:47:36] ah, yep, that'd do it :) [22:12:09] 10Continuous-Integration-Config, 10JavaScript: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [22:12:09] 10Continuous-Integration-Config, 10JavaScript: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [22:14:26] 10Continuous-Integration-Config, 10JavaScript: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [22:14:26] 10Continuous-Integration-Config, 10JavaScript: JS Linting: Drop `dot-notation` keyword over-ride from all repos; no longer needed - https://phabricator.wikimedia.org/T210552 (10Jdforrester-WMF) [23:04:32] 10Release-Engineering-Team, 10MediaWiki-API, 10Readers-Web-Backlog, 10Browser-Tests: Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (10Jdlrobson) [23:04:32] 10Release-Engineering-Team, 10MediaWiki-API, 10Readers-Web-Backlog, 10Browser-Tests: Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (10Jdlrobson) [23:04:40] 10Release-Engineering-Team, 10MediaWiki-API, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (10Jdlrobson) [23:04:40] 10Release-Engineering-Team, 10MediaWiki-API, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (10Jdlrobson) [23:05:37] 10Release-Engineering-Team, 10MediaWiki-API, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (10Jdlrobson)... [23:05:37] 10Release-Engineering-Team, 10MediaWiki-API, 10Browser-Tests, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q2): Ruby browser tests failing consistently with "The wiki is currently in read-only mode. (readonly) (MediawikiApi::ApiError)" - https://phabricator.wikimedia.org/T210557 (10Jdlrobson)... [23:05:49] Volker_E: on way! [23:05:49] Volker_E: on way! [23:30:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team, 10Patch-For-Review: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10bd808) All of the `10.*` references in hieradata/labs/deployment-prep/common.yaml are gone now, bu... [23:30:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team, 10Patch-For-Review: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10bd808) All of the `10.*` references in hieradata/labs/deployment-prep/common.yaml are gone now, bu... [23:34:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team, 10Patch-For-Review: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10Krenair) Yeah. There's some notes after relevant lines in https://etherpad.wikimedia.org/p/deploym... [23:34:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10cloud-services-team, 10Patch-For-Review: Ensure there are no hard-coded IPs in use for beta - https://phabricator.wikimedia.org/T208262 (10Krenair) Yeah. There's some notes after relevant lines in https://etherpad.wikimedia.org/p/deploym...