Sunday, April 9, 2017

GMime 2.99.0 released

After a long hiatus, I am pleased to announce the release of GMime 2.99.0!

See below for a list of new features and bug fixes.


About GMime

GMime is a C library which may be used for the creation and parsing of messages using the Multipurpose Internet Mail Extension (MIME), as defined by numerous IETF specifications.

GMime features an extremely robust high-performance parser designed to be able to preserve byte-for-byte information allowing developers to re-seralize the parsed messages back to a stream exactly as the parser found them. It also features integrated GnuPG and S/MIME v3.2 support.

Built on top of GObject (the object system used by the GNOME desktop), many developers should find its API design and memory management very familiar.


Noteworthy changes in version 2.99.0

  • Overhauled the GnuPG support to use GPGME under the hood rather than a custom wrapper.
  • Added S/MIME support, also thanks to GPGME.
  • Added International Domain Name support via GNU's libidn.
  • Improved the GMimeMessage APIs for accessing the common address headers. They now all return an InternetAddressList.
  • g_mime_init() no longer takes any flag arguments and the g_mime_set_user_charsets() API has also been dropped. Instead, GMimeParserOptions and GMimeFormatOptions have taken the place of these APIs to allow customization of various parser and formatting options in a much cleaner way. To facilitate this, many parsing functions and formatting functions have changed to now take these options arguments.
  • InternetAddress now has a 'charset' property that can be set to override GMime's auto-detection of the best charset to use when encoding names.
  • GMimeHeaderIter has been dropped in favor of a much simpler index-based API on GMimeHeaderList.
  • GMimeHeaderList no longer caches the raw message/mime headers in a stream. Instead, each GMimeHeader now has its own cache. This means that changing the GMimeHeaderList or any of its GMimeHeaders no longer invalidates the entire cache.
  • GMimeParser has been fixed to preserve (munged or otherwise) From-lines that sometimes appear at the start of the content of message/rfc822 parts.
  • GMimeParser now also scans for encapsulated PGP blocks within MIME parts as it is parsing them and sets a flag on each GMimePart that contains one of these blocks.
  • GMimePart now has APIs for dealing with said encapsulated PGP blocks.

Developers interested in migrating to the upcoming GMime 3.0 API (of which GMime 2.99.0 is a preview) should take a look at the PORTING document included with the source code as it contains a fairly comprehensive list of the API changes that they will need to be aware of.


Getting the Source Code

You can download official public release tarballs of GMime at https://download.gnome.org/sources/gmime/ or ftp://ftp.gnome.org/pub/GNOME/sources/gmime/.

If you would like to contribute to the GMime project, it is recommended that you grab the source code from the official GitHub repository at https://github.com/jstedfast/gmime. Cloning this repository can be done using the following command:

git clone https://github.com/jstedfast/gmime.git

Documentation

API reference documentation can be found at https://developer.gnome.org/gmime/2.99/.

Documentation for getting started can be found in the README.md.

MailKit 1.14 released

I am pleased to announce the release of MailKit 1.14!

See below for a list of new features and bug fixes.


About MailKit

MailKit is a C# library which is built on top of MimeKit and is intended to be used for interfacing with IMAP, POP3 and SMTP servers.

MailKit features incredibly robust IMAP, POP3 and SMTP clients with network APIs that are all capable of being canceled. API's that might transfer significant amounts of data between the client and server also include the ability to report progress. Asynchronous API's are also available.

Built on top of .NET, MailKit can be used with any of the .NET languages including C#, VB.NET, F#, and more. It will also run on any platform that Mono or the new .NET Core runtime have been ported to including Windows, Linux, Mac OS, Windows Phone, Apple TV, Apple Watch, iPhone/iPad, Xbox, PlayStation, and Android devices.


Noteworthy changes in version 1.14

  • Improved IMAP's BODYSTRUCTURE parser to sanitize the Content-Disposition values. (issue #486)
  • Improved robustness of IMAP's BODYSTRUCTURE parser in cases where qstring tokens have unescaped quotes. (issue #485)
  • Fixed IMAP to properly handle NIL as a folder name in LIST, LSUB and STATUS responses. (issue #482)
  • Added ImapFolder.GetHeaders() to allow developers to download the entire set of message headers.
  • Added SMTP support for International Domain Names in email addresses used in the MAIL FROM and RCPT TO commands.
  • Modified SmtpClient to no longer throw a NotSupportedException when trying to send messages to a recipient with a unicode local-part in the email address when the SMTP server does not support the SMTPUTF8 extension. Instead, the local-part is passed through as UTF-8, leaving it up to the server to reject either the command or the message. This seems to provide the best interoperability.

Installing via NuGet

The easiest way to install MailKit is via NuGet.

In Visual Studio's Package Manager Console, simply enter the following command:

Install-Package MailKit

Getting the Source Code

First, you'll need to clone MailKit from my GitHub repository. To do this using the command-line version of Git, you'll need to issue the following command in your terminal:

git clone --recursive https://github.com/jstedfast/MailKit.git

Documentation

API documentation can be found at http://mimekit.net/docs.

A copy of the xml formatted API documentation is also included in the NuGet and/or Xamarin Component package.

MimeKit 1.14 released

I am pleased to announce the release of MimeKit 1.14!

See below for a list of new features and bug fixes.


About MimeKit

MimeKit is a C# library which may be used for the creation and parsing of messages using the Multipurpose Internet Mail Extension (MIME), as defined by numerous IETF specifications.

MimeKit features an extremely robust high-performance parser designed to be able to preserve byte-for-byte information allowing developers to re-seralize the parsed messages back to a stream exactly as the parser found them. It also features integrated DKIM-Signature, S/MIME v3.2, OpenPGP and MS-TNEF support.

Built on top of .NET, MimeKit can be used with any of the .NET languages including C#, VB.NET, F#, and more. It will also run on any platform that Mono or the new .NET Core runtime have been ported to including Windows, Linux, Mac OS, Windows Phone, Apple TV, Apple Watch, iPhone/iPad, Xbox, PlayStation, and Android devices.


Noteworthy changes in version 1.14

  • Added International Domain Name support for email addresses.
  • Added a work-around for mailers that didn't provide a disposition value in a Content-Disposition header.
  • Added a work-around for mailers that quote the disposition value in a Content-Disposition header.
  • Added automatic key retrieval functionality for the GnuPG crypto context.
  • Added a virtual DigestSigner property to DkimSigner so that consumers can hook into services such as Azure. (issue #296)
  • Fixed a bug in the MimeFilterBase.SaveRemainingInput() logic.
  • Preserve munged From-lines at the start of message/rfc822 parts.
  • Map code page 50220 to iso-2022-jp.
  • Format Reply-To and Sender headers as address headers when using Header.SetValue().
  • Fixed MimeMessage.CreateFromMailMessage() to set the MIME-Version header. (issue #290)

Installing via NuGet

The easiest way to install MimeKit is via NuGet.

In Visual Studio's Package Manager Console, simply enter the following command:

Install-Package MimeKit

Getting the Source Code

First, you'll need to clone MimeKit from my GitHub repository. To do this using the command-line version of Git, you'll need to issue the following command in your terminal:

git clone --recursive https://github.com/jstedfast/MimeKit.git

Documentation

API documentation can be found at http://mimekit.net/docs.

A copy of the xml formatted API documentation is also included in the NuGet and/or Xamarin Component package.

Thursday, March 19, 2015

Code Review: Microsoft's System.Net.Mail Implementation

For those reading my blog for the first time and don't know who I am, allow myself to introduce... myself.

I'm a self-proclaimed expert on the topic of email, specifically MIME, IMAP, SMTP, and POP3. I don't proclaim myself to be an expert on much, but email is something that maybe 1 or 2 dozen people in the world could probably get away with saying they know more than I do and actually back it up. I've got a lot of experience writing email software over the past 15 years and rarely do I come across mail software that does things better than I've done them. I'm also a critic of mail software design and implementation.

My latest endeavors in the email space are MimeKit and MailKit, both of which are open source and available on GitHub for your perusal should you doubt my expertise.

My point is: I think my review carries some weight, or I wouldn't be writing this.

Is that egotistical of me? Maybe a little.

I was actually just fixing a bug in MimeKit earlier and when I went to go examine Mono's System.Net.Mail.MailMessage implementation in order to figure out what the problem was with my System.Net.Mail.MailMessage to MimeKit.MimeMessage conversion, I thought, "hey, wait a minute... didn't Microsoft just recently release their BCL source code?" So I ended up taking a look and pretty quickly confirmed my suspicions and was able to fix the bug.

When I begin looking at the source code for another mail library, I can't help but critique what I find.

MailAddress and MailAddressCollection


Parsing email addresses is probably the hardest thing to get right. It's what I would say makes or breaks a library (literally). To a casual onlooker, parsing email addresses probably seems like a trivial problem. "Just String.Split() on comma and then look for those angle bracket thingies and you're done, right?" Oh God, oh God, make the hurting stop. I need to stop here before I go into a long rant about this...

Okay, I'm back. Blood pressure has subsided.

Looking at MailAddressParser.cs (the internal parser used by MailAddressCollection), I'm actually pleasantly surprised. It actually looks pretty decent and I can tell that a lot of thought and care went into it. They actually use a tokenizer approach. Interestingly, they parse the string in reverse which is a pretty good idea, I must say. This approach probably helps simplify the parser logic a bit because parsing forward makes it difficult to know what the tokens belong to (is it the name token? or is it the local-part of an addr-spec? hard to know until I consume a few more tokens...).

For example, consider the following BNF grammar:

address         =       mailbox / group
mailbox         =       name-addr / addr-spec
name-addr       =       [display-name] angle-addr
angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr
group           =       display-name ":" [mailbox-list / CFWS] ";"
                        [CFWS]
display-name    =       phrase
word            =       atom / quoted-string
phrase          =       1*word / obs-phrase
addr-spec       =       local-part "@" domain
local-part      =       dot-atom / quoted-string / obs-local-part
domain          =       dot-atom / domain-literal / obs-domain
obs-local-part  =       word *("." word)

Now consider the following email address: "Joe Example" <joe@example.com>

The first token you read will be "Joe Example" and you might think that that token indicates that it is the display name, but it doesn't. All you know is that you've got a 'quoted-string' token. A 'quoted-string' can be part of a 'phrase' or it can be (a part of) the 'local-part' of the address itself. You must read at least 1 more token before you'll be able to figure out what it actually is ('obs-local-part' makes things slightly more difficult). In this case, you'll get a '<' which indicates the start of an 'angle-addr', allowing you to assume that the 'quoted-string' you just got is indeed the 'display-name'.

If, however, you parse the address in reverse, things become a little simpler because you know immediately what to expect the next token to be a part of.

That's pretty cool. Kudos to the Microsoft engineers for thinking up this strategy.

Unfortunately, the parser does not handle the 'group' address type. I'll let this slide, however, partly because I'm still impressed by the approach the address parser took and also because I realize that System.Net.Mail is meant for creating and sending new messages, not parsing existing messages from the wild.

Okay, so how well does it serialize MailAddress?

Ugh. You know that face you make when you just see a guy get kicked in the nuts? Yea, that's the face I made when I saw line #227:

encodedAddress = String.Format(CultureInfo.InvariantCulture, "\"{0}\"", this.displayName);

The problem with the above code (and I'll soon be submitting a bug report about this) is that the displayName string might have embedded double quotes in it. You can't just surround it with quotes and expect it to work. This is the same mistake all those programmers make that allow SQL-injection attacks.

For an example of how this should be done, see MimeKit's MimeUtils.Quote() method.

I had such high hopes... at least this is a fairly simple bug to fix. I'll probably just offer them a patch.

ContentType and ContentDisposition


Their parser is decent but it doesn't handle rfc2231 encoded parameter values, so I'm not overly impressed. It'll get the job done for simple name="value" parameter syntax, though, and it will decode the values encoded with the rfc2047 encoding scheme (which is not the right way to encode values, but it is common enough that any serious parser should handle it). The code is also pretty clean and uses a tokenizer approach, so that's a plus. I guess since this isn't really meant as a full-blown MIME parser, they can get away with this and not have it be a big deal. Fair enough.

Serialization, unsurprisingly, leaves a lot to be desired. Parameter values are, as I expected, encoded using rfc2047 syntax rather than the IETF standard rfc2231 syntax. I suppose that you could argue that this is for compatibility, but really it's just perpetuating bad practices. It also means that it can't properly fold long parameter values because the encoded value just becomes one big long encoded-word token. Yuck.

Base64


Amusingly, Microsoft does not use their Convert.FromBase64() decoder to decode base64 in their System.Net.Mail implementation. I point this out mostly because it is the single most common problem users have with every one of the Open Source .NET mail libraries out there (other than MimeKit, of course) because Convert.FromBase64() relies on the data not having any line breaks, white space, etc in the input stream.

This should serve as a big hint to you guys writing your own .NET email libraries not to use Convert.FromBase64() ;-)

They use unsafe pointers, just like I do in MimeKit, but I'm not sure how their performance compares to MimeKit's yet. They do use a state machine, though, so rock on.

I approve this base64 encoder/decoder implementation.

SmtpClient


One thing they do which is pretty cool is connection pooling. This is probably a pretty decent win for the types of things developers usually use System.Net.Mail's SmtpClient for (spam, anyone?).

The SASL AUTH mechanisms that they seem to support are NTLM, GSSAPI, LOGIN and WDIGEST (which apparently is some sort of IIS-specific authentication mechanism that I had never heard of until now). For those that were curious which SASL mechanisms SmtpClient supported, well, now you know.

The code is a bit hard to follow for someone not familiar with the codebase (not nearly as easy reading as the address or content-type parsers, I'm afraid), but it seems fairly well designed.

It does not appear to support PIPELINING or BINARYMIME like MailKit does, though. So, yay! Win for MailKit ;-)

They do support SMTPUTF8, so that's good.

It seems that if you set client.EnableSsl to true, it will also try STARTTLS if it isn't able to connect on the SSL port. I wasn't sure if it did that or not before, so this was something I was personally interested in knowing.

Hopefully my SmtpClient implementation review isn't too disappointing. I just don't know what to say about it, really. It's a pretty straight-forward send-command-wait-for-reply implementation and SMTP is pretty dead simple.

Conclusion


Overall the bits I was interested in were better than I expected they'd be. The parsers were pretty good (although incomplete) and the serializers were "good enough" for normal use.

Of course, it's not as good as MimeKit, but let's be honest, MimeKit sets the bar pretty high ;-)

Thursday, October 16, 2014

The Wait Is Over: MimeKit and MailKit Reach 1.0

After about a year in the making for MimeKit and nearly 8 months for MailKit, they've finally reached 1.0 status.

I started really working on MimeKit about a year ago wanting to give the .NET community a top-notch MIME parser that could handle anything the real world could throw at it. I wanted it to run on any platform that can run .NET (including mobile) and do it with remarkable speed and grace. I wanted to make it such that re-serializing the message would be a byte-for-byte copy of the original so that no data would ever be lost. This was also very important for my last goal, which was to support S/MIME and PGP out of the box.

All of these goals for MimeKit have been reached (partly thanks to the BouncyCastle project for the crypto support).

At the start of December last year, I began working on MailKit to aid in the adoption of MimeKit. It became clear that without a way to inter-operate with the various types of mail servers, .NET developers would be unlikely to adopt it.

I started off implementing an SmtpClient with support for SASL authentication, STARTTLS, and PIPELINING support.

Soon after, I began working on a Pop3Client that was designed such that I could use MimeKit to parse messages on the fly, directly from the socket, without needing to read the message data line-by-line looking for a ".\r\n" sequence, concatenating the lines into a massive memory buffer before I could start to parse the message. This fact, combined with the fact that MimeKit's message parser is orders of magnitude faster than any other .NET parser I could find, makes MailKit the fastest POP3 library the world has ever seen.

After a month or so of avoiding the inevitable, I finally began working on an ImapClient which took me roughly two weeks to produce the initial prototype (compared to a single weekend for each of the other protocols). After many months of implementing dozens of the more widely used IMAP4 extensions (including the GMail extensions) and tweaking the APIs (along with bug fixing) thanks to feedback from some of the early adopters, I believe that it is finally complete enough to call 1.0.

In July, at the request of someone involved with a number of the IETF email-related specifications, I also implemented support for the new Internationalized Email standards, making MimeKit and MailKit the first - and only - .NET email libraries to support these standards.

If you want to do anything at all related to email in .NET, take a look at MimeKit and MailKit. I guarantee that you will not be disappointed.

Monday, April 28, 2014

The Future of Debugging in Xamarin Studio

There comes a time in ever man's life when he says to himself, "there has got to be a better way..."

Set Next Statement

Have you ever been stepping through a method or hit a breakpoint and discovered that variables did not have the expected values? Don't you wish you could go back in time and start stepping through that method from an earlier point to see how things went so horribly wrong? Of course you do.

Last week, with the help of Zoltan Varga (who added the necessary runtime support), I implemented support in Xamarin Studio to set the next statement to execute when you resume execution of your program in the debugger. You can set the next statement to be any statement in the current method; any statement at all. This essentially allows you to jump back in time or completely step over execution of statements after the current position.

Don’t worry. As long as you hit Run To Cursor at precisely the moment the lightning strikes the tower, everything will be fine!

Run to Cursor

If you're like me, you've probably found yourself stepping through some code in the debugger and you get to a loop or something that you know is fine and you just don't feel like hitting Step Over the 5 bajillion times necessary to get past it, so what do you do? Hopefully you don't hit Step Over those 5 bajillion times. Hopefully you just set a breakpoint somewhere after that loop and then hit Continue.

The problem with this solution is that it's tedious.

Soon, however, you'll be able to simply right-click and select Run To Cursor (or just set/use a keybinding) and the debugger will resume execution until it reaches your cursor!

Client-Side Evaluation of Simple Properties

Assuming that you haven't disabled "Allow implicit property evaluation and method invocation" in your Xamarin Studio debugger preferences, whenever class properties are evaluated in the debugger (in the Watch pad, the Locals pad, or when you hover the mouse cursor over a property), in order to get the value, the debugger has to spin up a thread in the program being debugged in order to have it evaluate the property (or other expression) because, unlike fields, properties are really just methods that have to run arbitrary code.

For at least a year or so, now, we've mitigated this somewhat by cheating if the property has the CompilerGeneratedAttribute (signifying that it is an auto-property). When evaluating these properties, we would instead do a lookup of the backing field and get its value since we could do that locally without any need to round-trip to the program being debugged. While this helped a lot, there's a lot of properties out there that effectively just return a field and have no other logic (maybe an auto-property wasn't used because the setter does more than just set the field value?).

To improve performance of this, I started looking into what it would take to interpret the IL locally in the IDE. Obviously this could only really work if the property getter was "simple" enough and didn't have to take locks, etc. I started asking Zoltan some questions on the feasibility of this and he wrote a simple IL interpreter (included in Mono.Debugger.Soft) that I ended up using in Xamarin Studio to try and evaluate properties locally before falling back to having the debuggee's runtime spin up a thread to evaluate the property for us.

Great Scott! When Can I Start Using These Awesome Features?

To use these new features, you will need Mono 3.4.1 (or later) and an upcoming release of Xamarin Studio (5.0.1? It won't quite make it into 5.0).

Well, it's time I get back... to the future! ... And implementing more new features!

Code Snippet Licensing

All code posted to this blog is licensed under the MIT/X11 license unless otherwise stated in the post itself.