[03:19:51] <ryankemper>	 inflatador: codfw and eqiad done
[08:05:54] <dcausse>	 Caused by: java.time.format.DateTimeParseException: Text '2025-09-05T07:51:31.430+0000' could not be parsed, unparsed text found at index 23
[08:07:00] <dcausse>	 pfischer: the image_suggestion dag started to push tags but I think they ship a date format that is not supported
[08:49:57] <dcausse>	 gehel: if you have a moment https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/1185053 the SUP is currently blocked because of this
[10:02:36] <dcausse>	 lunch
[10:41:55] <dcausse>	 self merging
[11:10:29] <dcausse>	 sigh... Execution default of goal com.diffplug.spotless:spotless-maven-plugin:2.43.0:check failed: No such reference 'origin/main'
[11:10:33] <dcausse>	 https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/jobs/606770
[11:20:22] <dcausse>	 disabled for now in order to ship a fix for the search update pipeline
[12:11:30] <dcausse>	 the sup CI is kind of flaky failing with timeouts fetching devs from debian.org
[12:11:38] <dcausse>	 s/devs/debs
[12:23:04] <dcausse>	 the sup@eqiad was manually set to upgradeMode: stateless, not sure why, resetting to savepoint
[12:26:44] <dcausse>	 ok things are getting worse... now class org.apache.flink.api.java.typeutils.runtime.RowSerializer cannot be cast to class org.wikimedia.eventutilities.flink.EventRowSerializer (org.apache.flink.api.java.typeutils.runtime.RowSerializer is in unnamed module of loader 'app'; org.wikimedia.eventutilities.flink.EventRowSerializer is in unnamed module of loader
[12:26:46] <dcausse>	 org.apache.flink.util.ChildFirstClassLoader @516a6ef6)
[12:39:39] <gehel>	 dcausse: scream if you need help!
[12:41:00] <dcausse>	 very puzzled by what's going on, we should not have RowSerializer in the state...
[12:42:46] <gehel>	 I'm not going to be able to help much. Should you ask on Slack if someone can add 2 eyes and a brain to the discussion?
[12:43:38] <dcausse>	 sure, I will once I know what to ask :/
[12:44:45] <gehel>	 :)
[12:44:57] <gehel>	 by that time, you'll have the problem solved :)
[12:57:29] <dcausse>	 I feel that the state is borked...
[12:58:35] <dcausse>	 from a fresh state this works... going to assume that we were not doing the right before?
[13:08:36] <Trey314159>	 Anyone up for a review of https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1181781 ? The code is pretty simple.. the secret sauce, such as it is, is in the data.
[13:10:41] <dcausse>	 Trey314159: sure, I'll take a look on monday, was waiting for Peter to take a look and forgot about it :/
[13:11:04] <Trey314159>	 ahh.. makes sense.. thanks!
[13:17:08] <inflatador>	 <o/
[13:30:51] <inflatador>	 I'm verifying that we applied all the plugins from https://gitlab.wikimedia.org/repos/search-platform/opensearch-plugins-deb/-/merge_requests/7/diffs , does ` curl -s  $EP/_cat/plugins | grep opensearch-extra-analysis-khmer | grep -v wmf6` seem like a reasonable test?
[13:32:54] <inflatador>	 we're getting lots of `CirrusSearchNodeIndexingNotIncreasing` alerts in #data-platform-alerts, but then I don't see them in the alerts page. hmm
[13:33:04] <dcausse>	 inflatador: yes I'm on it
[13:33:32] <inflatador>	 dcausse ACK, ping me if I can help w/anything
[13:41:53] <ebernhardson>	 \o
[13:43:50] <dcausse>	 o/
[13:48:09] <inflatador>	 .o/
[13:52:12] <dcausse>	 destroyed the producer release and re-shipping
[13:52:19] <ebernhardson>	 sounds reasonable
[13:53:17] <dcausse>	 possibly what happened is when we upgraded to flink 1.20 we forgot to update to eventutilities to 1.4 (which has this: https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/1079506/11/eventutilities-flink/src/main/java/org/wikimedia/eventutilities/flink/EventRowTypeInfo.java)
[13:54:06] <dcausse>	 a new function to create the serializer and could have caused to fallback to RowSerializer instead of EventRowSerializer in the state
[13:54:21] <dcausse>	 use the createSerializer on the parent class
[13:54:52] <dcausse>	 this is kind of fragile... EventRowSerializer should probably not extend RowSerializer
[13:55:04] <dcausse>	 well... testing now, perhaps this is not that
[13:56:16] <dcausse>	 trying a restart to see if it can recover from its state
[13:56:25] <ebernhardson>	 sounds fragile indeed :S
[13:56:32] <dcausse>	 yes :(
[14:04:39] <ebernhardson>	 hmm, turns out can't convince phab paste to render a table :S
[14:13:31] <ebernhardson>	 a few examples of comparing query suggesters,  it's very not obvious from here so i guess i need to boil these down into numbers: https://phabricator.wikimedia.org/T403826
[14:13:57] <ebernhardson>	 we still do unfortunate rewrites like "motogp records all categories" -> "motown records all categories"
[14:16:43] <dcausse>	 and now it fails on restarts because it can't find its savepoints...
[14:16:59] <ebernhardson>	 :S
[14:25:22] <dcausse>	 it's not storing anything...
[15:00:11] <dcausse>	 it deletes its own savepoint on restart...
[15:01:41] <dcausse>	 INFO","message":"Disposing savepoint s3://cirrus-streaming-updater.wikikube-eqiad/producer/savepoints/savepoint-ce49a0-d83d9e7ac011."
[15:04:10] <ebernhardson>	 hmm, thats odd
[15:04:19] <ebernhardson>	 i don't understand why :S
[15:06:40] <ebernhardson>	 and thats after a full reset, which shouldn't need a savepoint?
[15:06:43] <dcausse>	 sigh https://issues.apache.org/jira/browse/FLINK-38033
[15:07:18] <dcausse>	 exactly what I'm seeing
[15:07:28] <ebernhardson>	 that does look to match :S
[15:07:41] <ebernhardson>	 so, downgrade to 1.11? Does that force the not-us projects to downgrade as well?
[15:08:40] <dcausse>	 or upgrade to 1.12.1
[15:09:16] <dcausse>	 but yes the current state is pretty bad... anyone deploying a new version might hit this bug
[15:09:23] <dcausse>	 :/
[15:09:50] <dcausse>	 not sure what to do...
[15:10:42] <dcausse>	 I can restart the sup and wait for monday to deploy a fixed version or 1.11
[15:10:43] <ebernhardson>	 hmm, it's the operator itself that needs to change? Backwards seems safest, but not sure what the implications are
[15:10:54] <ebernhardson>	 if it can run for the weekend, next week is certainly better than friday
[15:11:00] <dcausse>	 ok
[15:11:16] <dcausse>	 going to resume the SUP and file a task for monday
[15:11:46] <ebernhardson>	 +1
[15:11:46] <dcausse>	 could be pretty bad for the wdqs-updater... for search we can drop the state more easily
[15:16:34] <inflatador>	 should we warn gmodena- or anyone else who uses Flink?
[15:20:40] <dcausse>	 sure
[15:20:56] <dcausse>	 filing a ticket and will warn them on slack
[15:50:41] <inflatador>	 workout, back in ~40
[15:58:59] <dcausse>	 calling it a day, have a nice week-end
[16:24:57] <ebernhardson>	 extended regex seems to be working,  can search for `insource:/\n\n\n/` and indeed it finds pages with 3 sequential newlines
[16:25:58] <ebernhardson>	 highlighting is meh, but what can you expect when searching for unprintable chars :P
[16:33:42] <inflatador>	 back
[16:53:49] <inflatador>	 lunch, back in ~1h
[21:01:11] <inflatador>	 ryankemper you have anything for pairing? I've just been looking at the OpenSearch on K8s stuff
[21:03:01] <ryankemper>	 inflatador: mostly just tidying up gerrit patches. back home in 30, ping me if you’ve anything that needs to be looked at
[21:07:36] <inflatador>	 ryankemper np, I think we're good 'till next week. Enjoy the weekend!
[21:07:51] <ryankemper>	 likewise!