HA and Multi-Master Dovecot Setups #27
Comments
|
Hi @TCB13 Yes, I think we can both attest to the fact that some syncthing dev-team members clearly don't seem too happy about further discussions of dovecot specifics on the syncthing issues thread, despite the discussion really being in a syncthing context. One would expect the them to be a little more accommodating in light of the many different possible use case scenarios for their software, including this one, that is to be expected. At the same time, I suppose their response is much justified and understandable, so I am more than happy to oblige. This will indeed be a great place to further the discussion, so your contribution and any collaborations are most welcome :)
I have used unison before some years ago, for cross-syncing FTP config files, websites htdocs, etc between two servers (local development, and remote live production) for the previous company I worked at, with great success. It's a great tool, sharing similar features to both rsync and syncthing as you would already know. However, when I discovered syncthing, I quickly realized how much more powerful it was than unison, especially from the perspectives of automation, conflict-handling, and ease of setup/management. Needless to say, this made unison obsolete for me as I have never really looked back / found a need to use unison again ever since.
I've only really used OwnCloud, and that was some years ago to be honest. I'm looking to give NextCloud a try, as I believe it has come a long way since those earlier versions of OwnCloud. I found OwnCloud to be okay, just a bit too bloated in terms of the setup (Docker wasn't so popular yet then either), and also containing features that I didn't really need. It does a great job of simulating a fully-fledged Dropbox-like environment (besides for just simple file synchronization, also offering calendar & contact syncing, sharing/previewing of files, support for other plugins, etc). You make it sound like NextCloud has its own native mail server with built-in IMAP support and HA replication? If that is really the case, then I did not know about it and I have to say that does sound very promising. I'd need to research further before I can comment further on that
As I've already described in the Readme's for this project, one of the limitations of using a truly synchronous distributed file system for maildir replication, is precisely what you are referring to, namely the latency that is introduced when the filesystem engine tries to validate consistency of the replicated folder pairs between all notes. Syncthing as we know, inherently doesn't exhibit such behaviour In my case, one of the primary reasons that I wanted to set up a HA dovecot cluster in the first place, was to create my own E-mail "cloud", with servers distributed across the globe in key geographical areas, using geo-IP-based DNS resolution for the IMAP (and SMTP) hostname-to-ip lookups, allowing a more responsive experience for the IMAP client. For instance, I also use OfflineIMAP to sync my Gmail account to dovecot, as I find Gmail to be very sluggish over IMAP if using Google's IMAP servers as I think they shape the connection quite extensively. With this type of deployment, I can access and make extensive changes to my Gmail without any delays, allowing the server-related tasks of synchronization and replication to take place in the background, which greatly enhances my user experience when using an IMAP-based access instead of Google's own Gmail web interface. While I've never really used GlusterFS, my understanding is that it should handle the replication aspect fine, apart from the limitations we've alluded to, namely with a large number of files (which is typical in the case of mail servers), and the higher latency constraints - both of which you've rightly identified as well. My other concern is the dovecot cache, index, log files of course - but presumably, GlusterFS can exclude those too, in which case that problem would be solved... In conclusion, I think that with the advent of Syncthing - in particular that it is open source, multi-platform supported, lightweight, efficient, and reliable - there does not seem to be any need to venture into these other synchronization solutions, some of which aim to be a jack of all trades (and master of none) as in the case of NextCloud, or others which introduce their own set of limiting complications or issues which as we've discussed, make them less desirable for this specific use case But that's not to say that syncthing is perfect either. Already we've needed to make some modifications which (in some cases) might not be supported by the default operation, such as moving the temp files to the .stfolder, discarding of conflicts, etc. And there is no doubt room for further improvements as well. For instance, in terms of the dovecot UUID file: In a perfect replication scenario, no other changes should be able to replicate for a given directory unless the updated UUID file for that directory is transferred at exactly the same time.. While this is obviously not something that syncthing guarantees yet at this stage (or that I have implemented into my fork yet), practically speaking it has not been much of a problem either in my experience, as syncthing uses inode monitoring to invoke a sync, meaning that any changes are replicated almost instantaneously. In theory however, this could present a problem, as discrepancies in replication times between the syncing of dovecot UUID files vs actual mail file content, could "confuse" a dovecot node, forcing it to generate a new UUID mapping, which then in turn gets replicated back to all the other nodes, and invariably leads to the IMAP client needing to resync that whole folder from scratch. But dovecot seems to "self-heal" very reliably when that happens, without any destructive loss of actual mail data... and since that type of exception happens so rarely and under very specific conditions which in most cases would never even be provoked, it's not such a problem anyway! As I say, I think I've spent an abnormal amount of time experimenting with this, and tried all kinds of approaches before eventually coming right with this one, which has served me very well for some time now, to the point that, to great relief, I've not needed to look any further. When I first tried syncing maildirs, the big problems I came across were formation of blank or duplicate mails, and deleted mails reappearing. Eventually I figured out what was causing these phenomenon and managed to work around the causes, but that took a long time to figure out as I'm really more of a SysAdmin & NetAdmin than a software developer to be honest. But I do enjoy messing with this stuff, and so being able to deploy a working multi-master dovecot HA cluster that is entirely open source was an objective that I was determined to realise... I'm curious to know what type of solutions are used commercially. Proprietary software in many cases, I'd imagine? I know that dovecot does have a paid, commercial replication solution also which is not open source... but obviously that does not interest me. From the limited experiences I had using dsync, let's just say I'm not overly optimistic anyway... Hope this all made sense! I'd love to hear more from you, especially if you do find a better way of doing this :D Unless I've just been looking in all of the wrong places (and believe me when I say I've Googled it exhaustively), there does seem to be a lack of good resources/information on the web for this, after all. I think it's pretty awesome to come across others who share this same interest and passion and I look forward to seeing how we can develop this even further |
My apologies if I misslead you, NextCloud does not have it's own mail server. I was just linking to the setup because that guy managed to setup a replication architecture for NextCloud that is very close to what we can use with Dovecot.
Yes you're right, there isn't much information about this topic. It looks like the replication issues are "the hard stuff that nobody is willing to share much into what they're doing". I've tested the "official" dsync way to replicate with a 50 user test group and I wasn't happy with the results. It's too slow and you get random CPU spikes. You also get the limitation of operating in backend pairs, meaning this isn't a real scalable solution. Dovecot team actually says "The replication in general is a bit resource intensive, so it's not recommended to be used in multi-million user installations." so what can expect? :P According to the link bellow, Deutsche Telekom has a Dovecot setup that serves around 26 million users and their strategy is one of the following:
However I don't believe they're using Director do keep the IMAP clients connected to the right servers. Maybe some HAProxy based solution? Anyway, I also have an experimental setup running z-push. I'm not sure if you're familiar with the tech, but it's basically an open-source implementation of ActiveSync (with all they nice things) that runs in PHP and uses Dovecot's IMAP (or maildir directly) as a "storage backend". ActiveSync is an HTTP protocol meaning it's much easier to load balance and keep sessions on the right places than with IMAP. |
|
Designing a Planetary-Scale IMAP Service withConflict-free Replicated Data Types |
…ed to Xapian backported from github xapian issue #27 A Xapian term that begins with a capital letter, like "Jim", is actually treated by Xapian as the term "im" with the "J" prefix. Since Xapian interally stores data ignore case, we simply enforce the 1st letter to non-uppercase to inhibit the prefix processing.
I'm opening this issue as follow up to the original Syncthing issue where we discussed how to properly sync a Dovecot maildir without too much pain here: syncthing/syncthing#2208 (comment)
I've been researching / testing other interesting setups and found this:
An alternative to Syncthing, some people only seem to use it but I never tried it with a large setup. It also has the ability to watch the file system real time and do 2-way syncs. This tool is essentially a better and smarter version of rsync. Did you ever try this one?
Not related to Dovecot but the same strategies are applicable. This is probably one of the best ways to have a true HA multi-master setup since this fork takes care of the maildir issues.
I've tried this route before (not with Dovecot) but with web servers. It performs reasonable well until we get a large number of files to read - something that is bound to happen on a mail server. It seems like whenever you want to read a file GlusterFS tries to guarantee that the file still exists on all servers / replicas before serving the file. This makes reads very slow if you introduce high latency - multiple machines over WAN.
The text was updated successfully, but these errors were encountered: