Subject: [lemonade] Copy-via-catenate Considered Harmful

From: Philip Guenther <[email-masked]>
Date: Tue, 16 Aug 2005 01:01:00 -0700

In Paris, Dave Cridland mentioned that he had found it convenient
in his IMAP client to use CATENATE to copy messages without selecting
the source mailbox, specifying no TEXT chunks and but a single,
sectionless URL.  This bothered me at the time and I think I've
finally put my finger on why.

IMHO, using CATENATE to implement the 'copy a message' action is
the Wrong Thing for at least two reasons:

1) COPY preserves all the per-message state automatically; with
   CATENATE, the client has to first FETCH that info, then include
   it in the APPEND.  This implies that to truly copy a message,
   the client will need to select the mailbox so that it can fetch
   the current flags.

   And INTERNALDATE (if not cached).

   And annotations.  Oops, clients that don't support annotations
   but use copy-via-CATENATE will lose annotations.

   And what about the XFOOFOO data from that server extension your
   client doesn't know about?  So much for preserving state...

   Wait a moment, wasn't the whole point of copy-via-CATENATE that
   the client didn't need to select the mailbox?  If it's going to
   do that to fetch the flags and stuff, why not just use UID COPY
   and let the server sweat it out?  That requires less network
   traffic too.

2) "It is easier to optimize the specific than the general"
   COPY is both far older and far more specific than CATENATE.  As
   such, it is much more likely that a server will have an optimized
   COPY operation than an optimized CATENATE.  This is especially
   true for optimizations that depend on entire messages being
   copied.

   For example, consider 'black-box' servers that store each message
   in a separate file (e.g., Cyrus), where COPY can use OS-specific
   features such as link() to avoid the actual copying of message
   data.  To do that with CATENATE, the server would have to
    - notice that the CATENATE is of a single URL
    - that the URL is local
    - that the URL has no /;SECTION component
   Yeah, it's doable, but it's a chunk of code for a specific case
   that the client can already express more directly.  How long
   will it take for this to reach the top of the server implementor's
   "optimizations to add" list?

3) COPY may be able to do stuff that APPEND can't!  RFC 3503 ("MDN
   profile for IMAP"), section 4.3 suggests that "[Servers] SHOULD
   preserve the $MDNSent keyword on COPY even if the client does
   not have 'w' right."  No such recommendation is made for setting
   that flag with APPEND.

   In truth, this seems more like a bug in RFC 3503 than a real
   issue.  Still, the point stands.

Philip Guenther

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Dave Cridland <[email-masked]>
Date: Tue, 16 Aug 2005 11:16:09 +0100

As Philip alludes, my principle usage for this is to avoid the SELECT 
- it's not because of the network traffic of the SELECT itself, but 
because of the associated state change, which indirectly causes 
significant FETCH traffic.

In the general case, the source mailbox for a copy is SELECTed, 
since, in general, it's the one the user is looking at (literally, in 
the GUI), so I'm not sure this code-path ever gets exercised, but in 
principle, it could.

Where the source mailbox is SELECTed, then of course I use COPY - 
it's much more efficient - and if my client would need to fetch 
metadata to fulfill the COPY-via-CATENATE, then it certainly does 
issue a SELECT/COPY sequence, rather than 
SELECT/FETCH/APPEND-CATENATE, which would certainly be silly.

Actually more interesting is the case where the source mailbox is on 
another server, in which case the client issues FETCHes for 
individual body parts, and reassembles the message on the destination 
server, using BINARY at both ends as appropriate.

>    client doesn't know about?  So much for preserving state...
Sound points, and I certainly won't argue there.

Of course if you "edit" a draft, you lose all this metadata as well. 
And arguably, that's also a problem.

Consider a clever client interacting with a naïve client - the clever 
client creates the draft and adds various metadata, the naïve client 
edits, and the metadata is lost. I'm not trying to defend 
COPY-via-CATENATE, but it occurs to me that if it's a problem there, 
it could be a problem for the more traditional use-case of CATENATE.

>    and let the server sweat it out?  That requires less network
Not if you already have the metadata. If no SELECT would be required 
to perform the COPY-via-CATENATE, then there's markedly less traffic. 
Remember that SELECT causes all metadata to be marked stale, in 
effect, causing at the very least a CONDSTORE FETCH.

>    "optimizations to add" list?
I'm not inclined to take this seriously, sorry.

In the case where the client has sufficient metadata to perform 
CATENATE-via-COPY in its cache, there is significant gain to be made 
by avoiding the SELECT and its state change.

Yes, of course this may mean that COPY-via-CATENATE takes a handful 
of milliseconds extra, but the savings compared to the additional 
FETCHes far outweigh that extra latency. If CONDSTORE isn't 
supported, the extra SELECT can have very large traffic implications, 
if it is supported, it's still causing an extra command (and my 
client doesn't yet attempt to pipeline a CONDSTORE FETCH with the 
SELECT, hence it'll cause an extra round-trip, too - I'm still unsure 
if pipelining such a command would be entirely safe).

>    issue.  Still, the point stands.

This is, in some respects, a special-case of (1), so I wonder whether 
the same comments apply. Probably not, but perhaps it could be 
considered if we address the preservation of metadata mentioned in my 
comments to (1).

We could, for instance, state that:

Metadata, where not supplied, is copied from the first URL chunk 
which is used in the message's RFC2822 header, using the same rules 
as for COPY or UID COPY.

This would solve (1) and (3) for drafts as well as COPY-via-CATENATE. 
As I say, (2) is left as an exercise for the server implementor.

It also means that if you want to do a COPY but also change some 
metadata, it's possible to do that in one step.

Dave.

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 16 Aug 2005 11:01:53 -0400

Hi Dave,

How often is a 'copy' operation done as opposed to a 'move' (COPY then 
delete)? I would argue, based on what I see most users do, that a 'move' is 
done far more often than copy, and to do that a client has to 
SELECT/COPY/STORE - CATENATE won't work. Frankly I would have thought that 
the added complexity of supporting 'copy' using COPY-via-CATENATE and 
'move' using COPY/STORE would outweigh the benefits of using 
COPY-via-CATENATE.

You also make the assumption that any metadata the client has cached is the 
same as the state of the original message on the server. But you cannot say 
that for sure without doing a sync (SELECT/FETCH) on the original message 
before doing the actual CATENATE. So I think the assumption that you can 
rely on cached metadata to implement COPY-via-CATENATE is wrong for an 
environment where users may be using more than one client at a time.

-- 
Cyrus Daboo

---

From: Philip Guenther <[email-masked]>
Date: Tue, 16 Aug 2005 13:45:54 -0700

Dave Cridland <[email-masked]> writes:
...
>server, using BINARY at both ends as appropriate.

<offthread>
You reassemble the message from the individual body parts?  Depending on
how deep you run that process, that may break S/MIME signatures or
otherwise result in different content.  Consider the preamble and
epilogue on multiparts...

(It is apparently unclear whether S/MIME signatures cover the preamble
and epilogue if you sign a multipart.)
</offthread>

...
>effect, causing at the very least a CONDSTORE FETCH.

If you don't have the mailbox selected then your cache may already be
stale, you just don't know it yet.  Selecting the mailbox doesn't make
the data stale, _unselecting_ the mailbox does.  Selecting the mailbox
again and using CONDSTORE just happens to be the most efficient way of
refreshing.

>FETCHes far outweigh that extra latency.  If CONDSTORE isn't supported,
>the extra SELECT can have very large traffic implications,  <...>

Additional FETCHes?  I see an EXAMINE and a UID COPY, but no FETCHes.
Or are you saying that your client can't select a mailbox without
synchronizing?  If you find that slow, then don't do that: leave your
cache stale (your client is apparently fine with that already) and just
do the copy.

...
>command (and my client doesn't yet attempt to pipeline a CONDSTORE
>FETCH with the SELECT, hence it'll cause an extra round-trip, too - I'm
>still unsure if pipelining such a command would be entirely safe).

RFC 3501 says that "clients MUST NOT send multiple commands without
waiting if an ambiguity would result."  That clearly applies if the
connection is already in the selected state.  Whether it applies to a
client in the authenticated state isn't clear to me, though the consenus
seems to be that in practice, no server would reorder them.  You may get
a BAD from the FETCH if the SELECT/EXAMINE fails, but that's harmless.

...
>which is used in the message's RFC2822 header, using the same rules

Wouldn't the header be one of the more commonly replaced pieces of a
draft?  I know I often don't finalize the subject or recipient list of
my messages until after I finish composing the body.

At this point, I'm not convinced we know how this'll all be used well
enough to solve the apparent problem.

Philip Guenther

---

From: Dave Cridland <[email-masked]>
Date: Tue, 16 Aug 2005 23:20:15 +0100

> and epilogue if you sign a multipart.)

Gah. Oh, I hate crypto... Yes, currently I discard preambles and 
epilogues. I'll have to put in specific handling for S/MIME and 
potentially openpgp, then, probably copying the crypto part raw.

> again and using CONDSTORE just happens to be the most efficient way 

Ah, no, you're right, but you misunderstand. If I have a particular 
mailbox selected, then that is the mailbox I'm most likely to require 
being selected. Selecting another makes the data I'm actually working 
on stale, which means in this case, I'm looking at:

// bar selected
SELECT foo
COPY
SELECT bar
FETCH ...

That being what I meant.

> supported,

See above. :-) It's not that I can't select a mailbox without 
synchronizing (actually, I can, but normal API usage would synch the 
UID mapping, generally a 0-round-trip job), but that whatever I did 
have SELECTed is likely to need reselecting, and one or more FETCHes 
to resynch afterward.

> a BAD from the FETCH if the SELECT/EXAMINE fails, but that's 
Indeed, because if SELECT fails, it puts you back into the 
authenticated state. If a server does reorder commands, one assumes 
that it would not reorder across state changes.

I'm reasonably sure it's safe, but I've not thought it through 
enough, and I've not yet considered fast reselects as a suitable 
candidate for optimization quite yet, especially considering it'd 
only be sane for me with CONDSTORE, and that's per-mailbox, and could 
conceivably change during a session. Again, it should just generate a 
BAD.

> my messages until after I finish composing the body.
Hmmm... Good point. Unfortunately, that would involve resending the 
entire header, true.

> enough to solve the apparent problem.

I'm not even sure it *is* a problem.

I'm perfectly happy to remove COPY-via-CATENATE for the reasons both 
you and Cyrus have raised, although I suspect I'll leave it in for 
the single case where messages from multiple sources are being 
appended to a mailbox on a server which supports MULTIAPPEND, to 
maintain atomicity.

It occurs to me that some careful new syntax could replace the 
metadata stanza of the APPEND syntax with a "copy it from this 
source". But as I say, for the genuine, approved, usage of CATENATE, 
I'm not sure if this is a problem or not.

Dave.