399 lines
15 KiB
Plaintext
399 lines
15 KiB
Plaintext
Jabber Server Farming How-To
|
||
|
||
Ryan Eatmon
|
||
|
||
Jabber.org
|
||
|
||
reatmon@jabber.org
|
||
|
||
2002-06-06
|
||
Revision History
|
||
Revision 0.1 2002-06-06 Revised by: rwe
|
||
First draft.
|
||
|
||
|
||
Provide Jabber admins a look into the present and future of Jabber Server
|
||
Farming.
|
||
|
||
-----------------------------------------------------------------------------
|
||
Table of Contents
|
||
1. About this How-To
|
||
1.1. Purpose / Scope
|
||
1.2. About Jabber
|
||
1.3. Feedback
|
||
1.4. Copyrights and Trademarks
|
||
1.5. Acknowledgments and Thanks
|
||
|
||
|
||
2. Introduction to Jabberd and Farming
|
||
2.1. Jabberd - The Jabber.org Jabber Server
|
||
2.2. Farming
|
||
|
||
|
||
3. Implementing the Farm
|
||
3.1. Client-to-Server Farming w/ Round Robin DNS
|
||
3.2. Connection Redirection
|
||
3.3. Router Layer
|
||
3.4. Moving JSM State data into a DB
|
||
3.5. Multiple JSMs
|
||
3.6. JSM Communications
|
||
|
||
|
||
|
||
List of Figures
|
||
3-1. c2s Farming Diagram
|
||
|
||
-----------------------------------------------------------------------------
|
||
Chapter 1. About this How-To
|
||
|
||
1.1. Purpose / Scope
|
||
|
||
This document was started on June 06, 2002 by Ryan Eatmon to explain how to
|
||
set up a Jabber Server Farm using the Jabber.org Jabber Server ([http://
|
||
jabberd.jabberstudio.org] http://jabberd.jabberstudio.org).
|
||
|
||
This document is broken into two main sections. What is, and what will be.
|
||
As of the writing of this document the Jabber Server does not have all of the
|
||
pieces that are required to do full farming. There are steps that can be
|
||
taken to get partial farming, and those will be covered. There are also steps
|
||
that we are going to take, and those are covered to so that others can
|
||
contribute and comment.
|
||
-----------------------------------------------------------------------------
|
||
|
||
1.2. About Jabber
|
||
|
||
Jabber is an Open Source Instant Messaging System that uses XML as its base
|
||
protcol. I could spend an entire How-To describing Jabber, so I am just going
|
||
to point you to the Jabber.org web site. ([http://www.jabber.org] http://
|
||
www.jabber.org).
|
||
-----------------------------------------------------------------------------
|
||
|
||
1.3. Feedback
|
||
|
||
Comments on this How-To may be directed to the author (<reatmon@jabber.org
|
||
>).
|
||
-----------------------------------------------------------------------------
|
||
|
||
1.4. Copyrights and Trademarks
|
||
|
||
Copyright 2002 Ryan Eatmon
|
||
|
||
Permission is granted to copy, distribute and/or modify this document under
|
||
the terms of the GNU Free Documentation License, Version 1.1 or any later
|
||
version published by the Free Software Foundation; with no Invariant
|
||
Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of
|
||
the license is included in the appendix entitled "GNU Free Documentation
|
||
License".
|
||
|
||
Jabber is a Trademark of Jabber, Inc.
|
||
-----------------------------------------------------------------------------
|
||
|
||
1.5. Acknowledgments and Thanks
|
||
|
||
Thanks to everyone who gave me hints and tips on tuning Linux and
|
||
structuring of the farm. This includes Thomas Muldowney, Jeremie Miller, and
|
||
other members of the <jabberd@jabberstudio.org> list.
|
||
-----------------------------------------------------------------------------
|
||
|
||
Chapter 2. Introduction to Jabberd and Farming
|
||
|
||
2.1. Jabberd - The Jabber.org Jabber Server
|
||
|
||
The Jabberd server is a reference implementation for the Jabber protocol as
|
||
defined by the Jabber Software Foundation. It is written in C and is covered
|
||
by both the Jabber Open Source License(JOSL) and the GNU Public License(GPL).
|
||
The Jabberd project has its own documentation. ([http://
|
||
jabberd.jabberstudio.org/howto.html] http://jabberd.jabberstudio.org/
|
||
howto.html).
|
||
-----------------------------------------------------------------------------
|
||
|
||
2.2. Farming
|
||
|
||
This is an interesting topic. When you think of a server you tend to think
|
||
of a single machine running a single program. Farming is the idea of multiple
|
||
machines running multiple programs that all act together to appear to be one
|
||
machine/program.
|
||
|
||
Why farming? Two reasons with equal importance. Scalability and
|
||
Reliability.
|
||
|
||
Scalability is the idea of allowing something to handle many many more
|
||
connections/transactions at the same time. A typcial web server can handle
|
||
say 1000 connections a second, but 10 web servers acting together can provide
|
||
10,000 connections per second.
|
||
|
||
Reliability is the idea that even if a piece of the system goes down, the
|
||
system does not. There is no single point of failure, and there is built in
|
||
redundancy to ensure that something is there to pick up the slack if
|
||
something goes down. In the web server example, if one web server goes down
|
||
you could not send any connections to it until it is back up, thus ensuring
|
||
that your web site is always up.
|
||
|
||
Both of these ideas are crucial to a successful Farming strategy. Each
|
||
wants to push the design into a different direction, but they can be
|
||
reconciled into a single design that can scale and is reliable at the same
|
||
time.
|
||
-----------------------------------------------------------------------------
|
||
|
||
Chapter 3. Implementing the Farm
|
||
|
||
3.1. Client-to-Server Farming w/ Round Robin DNS
|
||
|
||
3.1.1. The Dream
|
||
|
||
The first thing that we can do is to split out the c2s from the server and
|
||
make it a seperate process/component. Once that is done, we can replicate the
|
||
c2s processes over multiple machines. Each one would connect back to the main
|
||
server running the JSM and everything should work just fine.
|
||
|
||
|
||
Figure 3-1. c2s Farming Diagram
|
||
|
||
[c2s-round-robin]
|
||
|
||
Pros:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>Easy to setup right now.
|
||
|
||
|
||
Cons:
|
||
|
||
<EFBFBD><EFBFBD>*<2A> Round Robin will not provide true load balancing since there is no
|
||
mechanism in place to check how many connections a server has when it
|
||
forwards a new connection to it the next time it comes up in the Round
|
||
Robin.
|
||
|
||
|
||
-----------------------------------------------------------------------------
|
||
3.1.2. The Reality
|
||
|
||
This is doable right now. The 1.5 version of jabberd has sought to break
|
||
all of the pieces out into seperate processes. This was not done for farming
|
||
specifically, but we will not complain since the jadc2s component can handle
|
||
upwards of 10k users. (The 1.4.x series c2s could only handle ~1024).
|
||
|
||
Currently there is a forked version of jadc2s that works with the 1.4.2
|
||
server. It is located in the jabberd14 CVS repository on JabberStudio. The
|
||
following example is running with two jadc2s boxes, and one central jabberd
|
||
box. To set this up do the following:
|
||
|
||
1. Get all of the [http://jabberd.jabberstudio.org/] source code for
|
||
jabberd14. Build the server, configure/make.
|
||
|
||
2. Get the [http://download.jabber.org/contrib/xdb_sql-1.2.tar.gz] source
|
||
for xdb_sql from [http://www.IDEALX.org] IDEALX , build it, and setup the
|
||
jabber.xml to use it.
|
||
|
||
Note This is a very important step. xdb_file, the default xdb that comes
|
||
with jabberd, is limited to open file descriptors too. You can play
|
||
the same shell games that we are going to play with jadc2s later,
|
||
but if you want a server that can handle millions or users, then you
|
||
need something other than xdb_file. Enter xdb_sql, which only uses
|
||
one file descriptor to connect to the mysql server.
|
||
|
||
For more information on how to configure xdb_sql, please see the README
|
||
that is distributed in the release.
|
||
|
||
3. Build jadc2s and distribute the binaries to the boxes where they will
|
||
run.
|
||
|
||
4. Setup the main jabberd to accept the jadc2s component connections. In
|
||
the jabber.xml config file, add the following XML:
|
||
<service id="jadc2s-1">
|
||
<accept>
|
||
<ip/>
|
||
<port>5111</port>
|
||
<secret>secret</secret>
|
||
</accept>
|
||
</service>
|
||
|
||
<service id="jadc2s-2">
|
||
<accept>
|
||
<ip/>
|
||
<port>5112</port>
|
||
<secret>secret</secret>
|
||
</accept>
|
||
</service>
|
||
|
||
|
||
Now you can run the main jabberd and get it listening for the jadc2s to
|
||
connect to it.
|
||
|
||
5. Configure the jadc2s.xml on each box to connect to the SM, where to
|
||
listen, etc...
|
||
<!-- session manager configuration -->
|
||
<sm>
|
||
<!-- host and port of the SM -->
|
||
<host>localhost</host>
|
||
<port>5111</port>
|
||
<!-- shared secret, for authenticating us -->
|
||
<secret>secret</secret>
|
||
|
||
<!-- our ID, for authenticating us to the sm -->
|
||
<id>jadc2s</id>
|
||
|
||
<!-- how many times to try to connect to the sm (default: 5) -->
|
||
<retries>5</retries>
|
||
</sm>
|
||
|
||
<!-- local network settings -->
|
||
<local>
|
||
<!-- who we identify ourselves as. This should correspond to the -->
|
||
<!-- ID (host) that the session manager thinks it is. -->
|
||
<id>localhost</id>
|
||
|
||
<!-- IP address to bind to (default: 0.0.0.0) -->
|
||
<!-- <ip>0.0.0.0</ip> -->
|
||
|
||
<!-- port to bind to (default: 5222) -->
|
||
<port>5222</port>
|
||
|
||
<!-- SSL configuration -->
|
||
<!-- Specify the port to listen on, and the key file to use for -->
|
||
<!-- the certificate. -->
|
||
<!-- <port/> (default: 5223) -->
|
||
<!-- <pemfile/> (default: ./server.pem) -->
|
||
<!--
|
||
<ssl>
|
||
<port>5223</port>
|
||
<pemfile>./server.pem</pemfile>
|
||
</ssl>
|
||
-->
|
||
</local>
|
||
|
||
|
||
For more information on how to configure jadc2s, please see the README
|
||
in the jadc2s source directory.
|
||
|
||
6. Open a shell where you can change file system parameters (root usually)
|
||
and execute the following command:
|
||
|
||
|
||
bash$ echo "24000" > /proc/sys/fs/file-max
|
||
|
||
|
||
This bumps the upper bound on the number of allowed file descriptors
|
||
that can be open at one time.
|
||
|
||
7. Set the limit for the shell you are in to use more than the default
|
||
1024 file descriptors.
|
||
|
||
|
||
bash$ ulimit -n 11000
|
||
|
||
|
||
|
||
8. Tell jadc2s how many file descriptors it is allowed to use:
|
||
<!-- maximum number of file descriptors. Should be a prime -->
|
||
<!-- number. At least four will be used by jadc2s itself, -->
|
||
<!-- usually around six or seven (default: 1023) -->
|
||
<!-- For a list of prime numbers: -->
|
||
<!-- http://www.utm.edu/research/primes/lists/small/10000.txt -->
|
||
<max_fds>10007</max_fds>
|
||
|
||
|
||
It is important that the number you pick is a prime number. To make it
|
||
easy to find the prime you want you can visit this page [http://
|
||
www.utm.edu/research/primes/lists/small/10000.txt] http://www.utm.edu/
|
||
research/primes/lists/small/10000.txt.
|
||
|
||
9. All that's left is to run the server, and the jadc2s processes.
|
||
Everything should work fine. If it doesn't, then PLEASE tell me at <
|
||
reatmon@jabber.org> so that I can fix this document.
|
||
|
||
|
||
-----------------------------------------------------------------------------
|
||
3.2. Connection Redirection
|
||
|
||
3.2.1. The Dream
|
||
|
||
Now that we have the c2s processes distributed across machines, we need to
|
||
get them to work together. We need two things, the ability to communicate how
|
||
many connections a c2s has to the others, and the ability to know who the
|
||
others are.
|
||
|
||
[22:21] Jer: well, the "protocol" could just be an auth error of 3xx for
|
||
redirect w/ the ip/port to connect to
|
||
|
||
Maybe we need to talk about the router layer sooner...
|
||
|
||
Pros:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>Load balances nicely.
|
||
|
||
|
||
Cons:
|
||
|
||
<EFBFBD><EFBFBD>*<2A> Requires client support for connection redirection. This is not hard to
|
||
do, it is just not part of any client right now.
|
||
|
||
|
||
-----------------------------------------------------------------------------
|
||
3.3. Router Layer
|
||
|
||
3.3.1. The Dream
|
||
|
||
|
||
|
||
Pros:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>
|
||
|
||
|
||
Cons:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>
|
||
|
||
|
||
-----------------------------------------------------------------------------
|
||
3.4. Moving JSM State data into a DB
|
||
|
||
3.4.1. The Dream
|
||
|
||
|
||
|
||
Pros:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>
|
||
|
||
|
||
Cons:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>
|
||
|
||
|
||
-----------------------------------------------------------------------------
|
||
3.5. Multiple JSMs
|
||
|
||
3.5.1. The Dream
|
||
|
||
|
||
|
||
Pros:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>
|
||
|
||
|
||
Cons:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>
|
||
|
||
|
||
-----------------------------------------------------------------------------
|
||
3.6. JSM Communications
|
||
|
||
3.6.1. The Dream
|
||
|
||
|
||
|
||
Pros:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>
|
||
|
||
|
||
Cons:
|
||
|
||
<EFBFBD><EFBFBD>*<2A>
|
||
|
||
|