ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation by clebertsuconic · Pull Request #6323 · apache/artemis

clebertsuconic · 2026-03-27T01:46:15Z

No description provided.

tabish121

Overall looks good, nice simplification of some of the message handling

...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java

Also making it immutable to avoid races after updates like we had in the past

...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java

tabish121 · 2026-04-07T19:00:30Z

...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java

+
+
+   // this is "borrowed" from:
+   // https://github.com/apache/qpid-proton-j/blob/6dc5587f1d1b23969a8994f1755198e638e92bc4/proton-j/src/main/java/org/apache/qpid/proton/codec/messaging/FastPathApplicationPropertiesType.java#L93-L115


This was inspired by the proton-j version but isn't a copy and it omits some of the actual validation checks that are done in the proton-j version which are not as thorough as they likely should be but are still better than not validating the encoding at all.

@tabish121 I don't understand what you're asking me to do here? update the comment?

@tabish121 this is being called right after the readConstructor was read. the only thing that would be decoded would be either this, or the skipValue() which is doing pretty much the same as this code is doing now.

I'm confused on what are you asking

I'm saying this does no validation on the data it reads, such as checking that the size value is smaller than the data remaining or on enforcing that the count is actually divisible by two which could indicate that the payload is corrupt if it isn't. I know proton-j does check at least the size data or maybe it checks that count isn't large than the data remaining in the buffer.

My point is that some validation might be sensible here to ensure valid data is being used up front.

This is doing the same validation thst was being done before.

If you follow the previous call all cobstructor.skipvalue does is read the value and move the bytes.

I can add the validation though, but this is the same semantic as before.

...ocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPStandardMessage.java

tabish121 · 2026-04-07T19:43:13Z

.../org/apache/activemq/artemis/tests/integration/amqp/AMQPDecodeApplicationPropertiesTest.java

+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class AMQPDecodeApplicationPropertiesTest {


Seems like a good place to add a test to validate that an AMQP message with no application properties actually has the correct values for the new size and count value tracking.

@tabish121 @gemmellr Do you guys know a way to write a NULL encoding, to validate this portion, without having me to deal with bytes directly? I was looking to add one on AMQPDecodeApplicationPropertiesTest.

https://github.com/apache/qpid-proton-j/blob/6dc5587f1d1b23969a8994f1755198e638e92bc4/proton-j/src/main/java/org/apache/qpid/proton/codec/messaging/FastPathApplicationPropertiesType.java#L111-L112

I'm not sure what null encoding you want to write. I was referring to a message that does not include the section ApplicationProperties at all, meaning it isn't encoded just a message like, Header, MessageAnnotations, Properties and an AmqpValue section

To write an ApplicationProperties that has no actual map payload that would be something like the following

buffer.writeByte((byte) 0); // Described Type Indicator buffer.writeByte(EncodingCodes.SMALLULONG); buffer.writeByte(ApplicationProperties.DESCRIPTOR_CODE.byteValue()); buffer.writeByte(EncodingCodes.NULL);

Proton MessageImpl encoding with an ApplicationProperties object assigned but that has not had a Map added via setValue would probably result in this encoding.

the codec, apparently support a value type that is empty. basically you can add EncodingCodes.NULL on the encoding and you get an empty ApplicationProperties.

I don't find it very useful but I see it on the codec.

As for the empty application properties I added a test.

I think we do have code somewhere in the tests that can encode messages with the structure you want, I don't recall where off the top of my head but again, it simply would be assigning an instance ApplicationProperties that has no body assigned to its setValue(Map) element and encoding it which should result in an null as the encoding for that part of the described type.

clebertsuconic · 2026-04-07T21:53:07Z

I added a commit to be squashed, returning the RELOAD state and persisting the memory estimate on the storage (journal or paging).

I honestly don't think the scan will make any difference.. The real issue is that the estimates were wrong in the first place...

But this commit should settle any discussion.

I think There are already version tests in the Version tests. but i will make sure about it soon.

clebertsuconic · 2026-04-07T22:57:57Z

for some reason I'm having to use a different persister, and I would have to digg on previous versions to understand why.

I will do some investigation tomorrow, but most likely I will need the V4 persister.

clebertsuconic · 2026-04-08T01:01:28Z

The reason I had to add a V4 is because of Paging:

On this part here, I encode the number of queues and the queueIDs:

artemis/artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/impl/PagedMessageImpl.java

Lines 227 to 234 in 5b12277

    
           int queueIDsSize = buffer.readInt(); 
        
           queueIDs = new long[queueIDsSize]; 
        
           for (int i = 0; i < queueIDsSize; i++) { 
        
              queueIDs[i] = buffer.readLong(); 
        
           }

At the point of the encoder I don't have any reference to the number of bytes used by its own decoder.

I'm adding one on V4 now, if in the future anything else is added, we will stop reading at that marker.

...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java

clebertsuconic · 2026-04-08T01:45:26Z

I intend to squash the second commit on the first as soon some review is done on the new persister.

clebertsuconic · 2026-04-08T14:58:00Z

I'm setting it as ready to review. but this commit should be squashed before merged.

clebertsuconic · 2026-04-08T16:34:53Z

I needed to add versioning to AMQPLargeMessagePersister as well.

I could choose to skip memory calculations on large message... however I will prefer to keep it the same way as StandardMessage.

this is also better for future additions

MemoryEstimate is now persisted with the message so we won't have to scan it to recalculate it. If loading a previous version, the server will at that point scan the message.

...l/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessagePersisterV4.java

tabish121 · 2026-04-08T21:59:27Z

...l/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessagePersisterV4.java

+      assert record != null && AMQPStandardMessage.class.equals(record.getClass());
+
+      // This might be useful in the future if we need to add more data to this persister. For now, it's just a filler.
+      int sizeV4 = buffer.readInt();


So what would you want to store in this extra int field that you've written the value if DataConstants.SIZE_INT which is four, so in the future you cannot write anything into there that can be four and know if you are reading current data or the old default. Would zero not be a better default as the absence of something is surely easier to deal with moving forward?

tabish121 · 2026-04-08T22:00:33Z

.../main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPLargeMessagePersisterV2.java

+      super.encode(buffer, record);
+
+      AMQPLargeMessage msgEncode = (AMQPLargeMessage) record;
+      buffer.writeInt(DataConstants.SIZE_INT);


Same comment as below here, writing 4 here means you've excluded the int value of four from having any meaning in this field moving forward

That's the number of bytes I'm currently writing in this persister.

I actually should limit
The buffer here. And skip bytes I didn't read.

I mean on the reading portion.

@tabish121 the idea is.. in the future, when I add another field, say, another Iteger, this becomes 8. so I know when to stop.

right now I can't do the magic comparison on buffer.hasReadableBytes() as I don't know how to delimit what belong to the persister.

However the reading portion is wrong as I should also skip for non reading values (think of inverted compatibility, from newer to odler version)

I see what you are doing, the comments did not make that at all clear though so maybe improve them a bit?

@tabish121 I added a commit with a code change here

tabish121 · 2026-04-08T22:26:54Z

...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java

    */
   public enum MessageDataScanningStatus {
-      NOT_SCANNED(0), SCANNED(1);
+      NOT_SCANNED(0), RELOAD_PERSISTENCE(1), SCANNED(2);


So since you are now storing the memory estimate in the V4 persistent form can you not alter recoverHeaderDataFromEncoding to provide it the version you are recovering from if less than V4 you also scan and load the values from the encoded ApplicationPropreties if present (using the existing parseAndSkip method) and thereby remove the need for the RELOAD_PERSISTENCE state that you wanted to remove. The code would just need to know when to stop scanning which would be either as soon as a Header was read, or under a similar set of constraints as scanMessageData where it stops after having hit the body sections or a Footer

I could add anything to not needed parsing... but I'm confused on what should I write.

You will stil need the code in place here for version compatbility I guess... (or I guess we could just do a full scan if from an older version.

Ohhhh... this is just to recover the priority out of the Header.

Sure I can write the priority as part of the persister.

@tabish121 I pushed a commit that completely eliminates the scanning during reload.

I could remove the RELOAD status (still thinking about it.. it's only needed for isPersistent now. I could persist it on the persister as well and just have it NOT SCANNED.

Please check it out..

(these commits will need to be squashed)

One thing I will work on this afternoon is to add a checker for LargeMessages as well on AMQPReloadFromPersistenceTest

clebertsuconic force-pushed the message-size-amqp branch 8 times, most recently from f70cda5 to be90ac4 Compare March 29, 2026 21:44

clebertsuconic changed the title ~~ARTEMIS-TBD Improving memory estimates on AMQP~~ ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation and make it static Mar 29, 2026

clebertsuconic force-pushed the message-size-amqp branch 9 times, most recently from 0611d5a to 03b7176 Compare March 30, 2026 22:51

clebertsuconic marked this pull request as ready for review March 30, 2026 22:51

clebertsuconic force-pushed the message-size-amqp branch 8 times, most recently from 94efe35 to 70beb3a Compare April 1, 2026 13:07

clebertsuconic changed the title ~~ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation and make it static~~ ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation Apr 1, 2026

tabish121 self-requested a review April 1, 2026 14:14

tabish121 reviewed Apr 1, 2026

View reviewed changes

...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java Outdated Show resolved Hide resolved

...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java Outdated Show resolved Hide resolved

clebertsuconic force-pushed the message-size-amqp branch from 7c02c99 to 8f1e497 Compare April 7, 2026 12:36

clebertsuconic commented Apr 7, 2026

View reviewed changes

...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java Show resolved Hide resolved

ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation

6606896

Also making it immutable to avoid races after updates like we had in the past

clebertsuconic force-pushed the message-size-amqp branch from 8f1e497 to 6606896 Compare April 7, 2026 12:49

tabish121 reviewed Apr 7, 2026

View reviewed changes

clebertsuconic marked this pull request as draft April 7, 2026 21:51

clebertsuconic force-pushed the message-size-amqp branch 7 times, most recently from 09bb86b to 540ad20 Compare April 8, 2026 00:58

clebertsuconic force-pushed the message-size-amqp branch 4 times, most recently from 1cb4403 to fa1128e Compare April 8, 2026 01:26

clebertsuconic commented Apr 8, 2026

View reviewed changes

...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java Outdated Show resolved Hide resolved

ARTEMIS-5573 and ARTEMIS-5975 Adding RELOAD back

6889336

MemoryEstimate is now persisted with the message so we won't have to scan it to recalculate it. If loading a previous version, the server will at that point scan the message.

clebertsuconic commented Apr 8, 2026

View reviewed changes

...l/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessagePersisterV4.java Outdated Show resolved Hide resolved

tabish121 reviewed Apr 8, 2026

View reviewed changes

clebertsuconic added 2 commits April 8, 2026 18:38

limiting reading for future version... addressing comment

dc062a5

No more scanning at all

ab62605



		// this is "borrowed" from:
		// https://github.com/apache/qpid-proton-j/blob/6dc5587f1d1b23969a8994f1755198e638e92bc4/proton-j/src/main/java/org/apache/qpid/proton/codec/messaging/FastPathApplicationPropertiesType.java#L93-L115


		import static org.junit.jupiter.api.Assertions.assertEquals;

		public class AMQPDecodeApplicationPropertiesTest {

Conversation

clebertsuconic commented Mar 27, 2026

Uh oh!

tabish121 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tabish121 Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clebertsuconic Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clebertsuconic commented Apr 7, 2026

Uh oh!

clebertsuconic commented Apr 7, 2026

Uh oh!

clebertsuconic commented Apr 8, 2026

Uh oh!

Uh oh!

clebertsuconic commented Apr 8, 2026

Uh oh!

clebertsuconic commented Apr 8, 2026

Uh oh!

clebertsuconic commented Apr 8, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

tabish121 Apr 8, 2026 •

edited

Loading

clebertsuconic Apr 8, 2026 •

edited

Loading