I'm writing a plugin in thunderbird using native messaging (following the ping pong example in python) to call a Delphi program to copy an e-mail locally as an ".eml" file. The problem I am facing seems to be the encoding. In addition, the resulting file contains double quotes ("") at the start and the end of the file as well as escaped double quotes (\"). I just want to have a 1 to 1 copy and not to change its content.
Example of a mail content:
"test"
€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ
éöàäèüâêû
However, in the file, it looks more like this:
\"test\"
€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“â€â€¢â€“—˜™š›œžŸ
éöà äèüâêû
I might have found the problem, which is explained here:
https://www.i18nqa.com/debug/utf8-debug.html
However, I do not really know how to adapt my code to solve this problem.
Thank you for your help!
Here is my background.js:
async function main() {
messenger.menus.create({
contexts : ["message_list"],
id: "copy@mail.lu",
onclick : passMsg,
title: messenger.i18n.getMessage("lang.menuTitle")
});
}
async function passMsg(OnClickData) {
if (OnClickData.selectedMessages && OnClickData.selectedMessages.messages.length > 0) {
let MessageHeader = OnClickData.selectedMessages.messages[0];
let raw = await messenger.messages.getRaw(MessageHeader.id);
let port=browser.runtime.connectNative("copymail");
port.onMessage.addListener((message) => {
port.disconnect();
});
port.postMessage(raw);
} else {
console.log("No message selected");
}
}
main();
Here is my Delphi code:
procedure WriteSTDInputToFile(const Filename: String);
var
Buffer: array [0 .. 3] of Byte;
msgLen: LongInt;
msg: UTF8String;
myFile: TextFile;
StdIn: THandleStream;
jsonValue: TJSONValue;
begin
StdIn := THandleStream.Create(GetStdHandle(STD_INPUT_HANDLE));
try
msgLen := 0;
if StdIn.Read(Buffer, SizeOf(msgLen)) > 0 then
msgLen := PLongInt(@Buffer)^;
if msgLen > 0 then
begin
SetLength(msg, msgLen);
StdIn.Read(PUTF8Char(msg)^, msgLen);
if msg <> '' then
begin
AssignFile(myFile, Filename, CP_UTF8);
ReWrite(myFile);
jsonValue := TJSONObject.ParseJSONValue(msg);
try
write(myFile, UTF8Encode(jsonValue.ToString));
finally
jsonValue.Free;
end;
CloseFile(myFile);
end;
end;
finally
if Assigned(StdIn) then
StdIn.Free;
end;
end;
Resulting file content:
"X-MDAV-Result: clean
X-MDAV-Processed: mail.test.lu, Wed, 28 Oct 2020 08:13:22 +0100
X-Spam-Processed: mail.test.lu, Wed, 28 Oct 2020 08:13:22 +0100
Return-path: <copy@mail.lu>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on MAIL01E
X-Spam-Level:
X-Spam-Status: No, score=0.7 required=10.0 tests=HTML_MESSAGE,MPART_ALT_DIFF
shortcircuit=no autolearn=disabled version=3.4.2
Authentication-Results: test.lu;
auth=pass (plain) smtp.auth=ascholtes@test.lu
Received: from [172.16.17.35] [(172.16.17.35)] by test.lu (172.31.3.6) with ESMTPSA id md50033234892.msg;
Wed, 28 Oct 2020 08:13:21 +0100
X-MDRemoteIP: 172.16.17.35
X-MDArrival-Date: Wed, 28 Oct 2020 08:13:21 +0100
X-Authenticated-Sender: ascholtes@test.lu
X-Rcpt-To: copy@mail.lu
X-MDRcpt-To: copy@mail.lu
X-Return-Path: copy@mail.lu
X-Envelope-From: copy@mail.lu
X-MDaemon-Deliver-To: ascholtes@test.lu
To: Ayuth Scholtes <copy@mail.lu>
From: Ayuth Scholtes <copy@mail.lu>
Subject: Test
Organization: CISS
Message-ID: <7eb36f7c-a7af-c272-c189-eded642c3e1c@test.lu>
Date: Wed, 28 Oct 2020 08:13:21 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101
Thunderbird/68.10.0
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary=\"------------6068A746223BB2C9F1771938\"
Content-Language: lb-LU
This is a multi-part message in MIME format.
--------------6068A746223BB2C9F1771938
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
|\"test\" â¬âÆââ¦â â¡Ëâ°Å â¹ÅŽâââââ¢ââËâ¢Å¡âºÅžŸ éöà äèüâêû|
--------------6068A746223BB2C9F1771938
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv=\"content-type\" content=\"text/html; charset=UTF-8\">
</head>
<body>
<pre class=\"lang-pascal s-code-block hljs delphi\"><code>\"test\"
â¬âÆââ¦â â¡Ëâ°Å â¹ÅŽâââââ¢ââËâ¢Å¡âºÅžŸ
éöà äèüâêû</code></pre>
</body>
</html>
--------------6068A746223BB2C9F1771938--
"
At first let me say that you did a good job in transferring data between web extension (Thunderbird add-on) and native application using native messaging. It isn't easy to understand it and set it up, but you managed to transfer required data with some tiny glitches you describe in your question.
... the resulting file contains double quotes (
") at the start and the end of the file as well as escaped double quotes (\")
In the add-on you obtain raw email data as a string - console.log(typeof raw) gives string which you then pass to port.postMessage. Although the documentation says it takes JSON object representing the message to send, but it seems to accept single string value which is valid JSON according to some standards. In Delphi code you receive the message via STDIN and parse it using TJSONObject.ParseJSONValue into TJSONValue. It will in fact create instance of TJSONString. You can verify that by examining the value of jsonValue.ClassName. The problem with quotes arises when you use jsonValue.ToString which returns quoted version of the string that is basically the same what you had before parsing. Use the Value property to return raw string value.
Using jsonValue.Value alone will not help you with the encoding issue. The raw message data that you obtain from the e-mail client is in EML format. It conforms to RFC-822 and that means it is ASCII encoded, but it can contain arbitrarily encoded message parts (see your own sample EML). Since you only want to save EML file as is not taking any encoding into account, the best would be to transfer raw bytes of EML, but this isn't out-of-the-box supported by Javascript and native messaging API. Therefore I'd suggest you to send Base64-encoded data string to native application where you decode it into raw bytes that you can write straight to disk.
To encode raw message data as Base64 string in add-on use function btoa:
port.postMessage(btoa(raw));
To receive the message in native application you can do the following:
uses
System.SysUtils, System.Classes, System.IOUtils, System.JSON, System.NetEncoding, Winapi.Windows;
procedure WriteSTDInputToFile(const FileName: string);
var
StdIn: THandleStream;
MsgLen: Cardinal;
Data: TBytes;
JSONValue: TJSONValue;
begin
StdIn := THandleStream.Create(GetStdHandle(STD_INPUT_HANDLE));
try
StdIn.ReadBuffer(MsgLen, SizeOf(MsgLen));
SetLength(Data, MsgLen);
StdIn.ReadBuffer(Data, MsgLen);
JSONValue := TJSONObject.ParseJSONValue(Data, 0);
Data := TNetEncoding.Base64.DecodeStringToBytes(JSONValue.Value);
TFile.WriteAllBytes(FileName, Data);
finally
StdIn.Free;
end;
end;
Notice several improvements to the original code:
Cardinal type for MsgLen. The protocol defines that the first 4 bytes on input indicate message length in bytes represented as 32-bit unsigned integer. Cardinal is Delphi's native type for such a value, or you can use UInt32 alias as well.ReadBuffer method instead of Read to read from STDIN, which makes the program crash in case of some unexpected circumstances. Ideally you should handle such circumstances, send error message in response via STDOUT and handle the response in add-on.File.WriteAllBytes from System.IOUtils.if Assigned(StdIn) then StdIn.Free;. This is what Free already does for you.Knowing that the incoming message is a quoted Base64-encoded string it would be possible to leave out JSON processing, so that the code becomes:
procedure WriteSTDInputToFile(const FileName: string);
var
StdIn: THandleStream;
MsgLen: Cardinal;
Msg: RawByteString;
Data: TBytes;
begin
StdIn := THandleStream.Create(GetStdHandle(STD_INPUT_HANDLE));
try
StdIn.ReadBuffer(MsgLen, SizeOf(MsgLen));
StdIn.Seek(1, soFromCurrent); { skip double quote }
SetLength(Msg, MsgLen - 2); { minus leading and trailing double quotes }
StdIn.ReadBuffer(Msg[Low(Msg)], MsgLen);
Data := TNetEncoding.Base64.DecodeStringToBytes(UTF8ToString(Msg));
TFile.WriteAllBytes(FileName, Data);
finally
StdIn.Free;
end;
end;