mirror of https://github.com/tLDP/LDP
556 lines
22 KiB
XML
556 lines
22 KiB
XML
<?xml version='1.0' encoding='ISO-8859-1'?>
|
|
<!DOCTYPE book PUBLIC '-//OASIS//DTD DocBook XML V4.1.2//EN'
|
|
"/usr/share/sgml/docbook/dtd/xml/4.1.2/docbookx.dtd" [
|
|
|
|
]>
|
|
|
|
|
|
<book id="index">
|
|
<bookinfo>
|
|
<title>Jabber Server Farming How-To</title>
|
|
<pubdate>2002-06-06</pubdate>
|
|
<author>
|
|
<firstname>Ryan</firstname>
|
|
<surname>Eatmon</surname>
|
|
<affiliation>
|
|
<orgname>Jabber.org</orgname>
|
|
<address><email>reatmon@jabber.org</email></address>
|
|
</affiliation>
|
|
</author>
|
|
|
|
<abstract>
|
|
<para>Provide Jabber admins a look into the present and future of Jabber Server Farming.</para>
|
|
</abstract>
|
|
<revhistory id="revhistory">
|
|
<revision>
|
|
<revnumber>0.1</revnumber>
|
|
<date>2002-06-06</date>
|
|
<authorinitials>rwe</authorinitials>
|
|
<revremark>First draft.</revremark>
|
|
</revision>
|
|
</revhistory>
|
|
</bookinfo>
|
|
|
|
<chapter id="aboutthishowto">
|
|
<title>About this How-To</title>
|
|
|
|
<section id='purpose'>
|
|
<title>Purpose / Scope</title>
|
|
<para>
|
|
This document was started on June 06, 2002 by Ryan Eatmon to
|
|
explain how to set up a Jabber Server Farm using the Jabber.org
|
|
Jabber Server (<ulink url='http://jabberd.jabberstudio.org'>http://jabberd.jabberstudio.org</ulink>).
|
|
</para>
|
|
<para>
|
|
This document is broken into two main sections. What is, and
|
|
what will be. As of the writing of this document the Jabber
|
|
Server does not have all of the pieces that are required to
|
|
do full farming. There are steps that can be taken to get
|
|
partial farming, and those will be covered. There are also
|
|
steps that we are going to take, and those are covered to so
|
|
that others can contribute and comment.
|
|
</para>
|
|
</section>
|
|
|
|
<section id='aboutjabber'>
|
|
<title>About Jabber</title>
|
|
<para>
|
|
Jabber is an Open Source Instant Messaging System that uses
|
|
XML as its base protcol. I could spend an entire How-To
|
|
describing Jabber, so I am just going to point you to the
|
|
Jabber.org web site. (<ulink url='http://www.jabber.org'>http://www.jabber.org</ulink>).
|
|
</para>
|
|
</section>
|
|
|
|
<section id="feedback">
|
|
<title>Feedback</title>
|
|
<para>
|
|
Comments on this How-To may be directed to the author
|
|
(<email>reatmon@jabber.org</email>).
|
|
</para>
|
|
</section>
|
|
|
|
<section id="copyrights">
|
|
<title>Copyrights and Trademarks</title>
|
|
<para>Copyright 2002 Ryan Eatmon</para>
|
|
<para>
|
|
Permission is granted to copy, distribute and/or modify
|
|
this document under the terms of the GNU Free Documentation
|
|
License, Version 1.1 or any later version published by
|
|
the Free Software Foundation; with no Invariant Sections,
|
|
with no Front-Cover Texts, and with no Back-Cover Texts.
|
|
A copy of the license is included in the appendix entitled
|
|
"GNU Free Documentation License".
|
|
</para>
|
|
<para>Jabber™ is a Trademark of Jabber, Inc.</para>
|
|
</section>
|
|
|
|
<section id="acknowledgements">
|
|
<title>Acknowledgments and Thanks</title>
|
|
<para>
|
|
Thanks to everyone who gave me hints and tips on tuning
|
|
Linux and structuring of the farm. This includes Thomas
|
|
Muldowney, Jeremie Miller, and other members of the
|
|
<email>jabberd@jabberstudio.org</email> list.
|
|
</para>
|
|
</section>
|
|
</chapter>
|
|
|
|
|
|
<chapter id='introduction'>
|
|
<title>Introduction to Jabberd and Farming</title>
|
|
|
|
<section id='jabberserver'>
|
|
<title>Jabberd - The Jabber.org Jabber Server</title>
|
|
<para>
|
|
The Jabberd server is a reference implementation for
|
|
the Jabber protocol as defined by the Jabber Software
|
|
Foundation. It is written in C and is covered by both
|
|
the Jabber Open Source License(JOSL) and the GNU Public
|
|
License(GPL). The Jabberd project has its own
|
|
documentation. (<ulink url='http://jabberd.jabberstudio.org/howto.html'>http://jabberd.jabberstudio.org/howto.html</ulink>).
|
|
</para>
|
|
</section>
|
|
|
|
<section id='farming'>
|
|
<title>Farming</title>
|
|
|
|
<para>
|
|
This is an interesting topic. When you think of a
|
|
server you tend to think of a single machine running
|
|
a single program. Farming is the idea of multiple machines
|
|
running multiple programs that all act together to
|
|
appear to be one machine/program.
|
|
</para>
|
|
<para>
|
|
Why farming? Two reasons with equal importance. Scalability
|
|
and Reliability.
|
|
</para>
|
|
<para>
|
|
Scalability is the idea of allowing something to handle
|
|
many many more connections/transactions at the same time.
|
|
A typcial web server can handle say 1000 connections a
|
|
second, but 10 web servers acting together can provide
|
|
10,000 connections per second.
|
|
</para>
|
|
<para>
|
|
Reliability is the idea that even if a piece of the system
|
|
goes down, the system does not. There is no single point
|
|
of failure, and there is built in redundancy to ensure
|
|
that something is there to pick up the slack if something
|
|
goes down. In the web server example, if one web server
|
|
goes down you could not send any connections to it until
|
|
it is back up, thus ensuring that your web site is always
|
|
up.
|
|
</para>
|
|
<para>
|
|
Both of these ideas are crucial to a successful Farming
|
|
strategy. Each wants to push the design into a different
|
|
direction, but they can be reconciled into a single
|
|
design that can scale and is reliable at the same time.
|
|
</para>
|
|
</section>
|
|
</chapter>
|
|
|
|
<chapter id='implementation'>
|
|
<title>Implementing the Farm</title>
|
|
|
|
<section id='c2sfarm'>
|
|
<title>Client-to-Server Farming w/ Round Robin DNS</title>
|
|
|
|
<section id='c2sfarmdream'>
|
|
<title>The Dream</title>
|
|
|
|
<para>
|
|
The first thing that we can do is to split out
|
|
the c2s from the server and make it a seperate
|
|
process/component. Once that is done, we can
|
|
replicate the c2s processes over multiple
|
|
machines. Each one would connect back to the
|
|
main server running the JSM and everything
|
|
should work just fine.
|
|
</para>
|
|
<figure>
|
|
<title>c2s Farming Diagram</title>
|
|
<mediaobject>
|
|
<imageobject>
|
|
<imagedata fileref="c2s-round-robin.ps" format="PS" />
|
|
</imageobject>
|
|
<imageobject>
|
|
<imagedata fileref="c2s-round-robin.jpeg" format="JPG" />
|
|
</imageobject>
|
|
<textobject>
|
|
<phrase>Diagram of c2s using Round Robin for Farming.</phrase>
|
|
</textobject>
|
|
</mediaobject>
|
|
</figure>
|
|
|
|
<para>Pros:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>Easy to setup right now.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para>Cons:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
Round Robin will not provide true load
|
|
balancing since there is no mechanism in
|
|
place to check how many connections a
|
|
server has when it forwards a new connection
|
|
to it the next time it comes up in the
|
|
Round Robin.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
|
|
<section id='c2sfarmreality'>
|
|
<title>The Reality</title>
|
|
<para>
|
|
This is doable right now. The 1.5 version of
|
|
jabberd has sought to break all of the pieces
|
|
out into seperate processes. This was not
|
|
done for farming specifically, but we will not
|
|
complain since the jadc2s component can handle
|
|
upwards of 10k users. (The 1.4.x series c2s
|
|
could only handle ~1024).
|
|
</para>
|
|
<para>
|
|
Currently there is a forked version of jadc2s
|
|
that works with the 1.4.2 server. It is located
|
|
in the jabberd14 CVS repository on JabberStudio.
|
|
The following example is running with two jadc2s
|
|
boxes, and one central jabberd box. To set this
|
|
up do the following:
|
|
</para>
|
|
<orderedlist>
|
|
<listitem>
|
|
<para>
|
|
Get all of the
|
|
<ulink url='http://jabberd.jabberstudio.org/'>source code for jabberd14</ulink>.
|
|
Build the server, configure/make.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Get the
|
|
<ulink url='http://download.jabber.org/contrib/xdb_sql-1.2.tar.gz'>source</ulink>
|
|
for xdb_sql from
|
|
<ulink url='http://www.IDEALX.org'>IDEALX</ulink>
|
|
, build it, and setup the jabber.xml to use it.
|
|
</para>
|
|
<note>
|
|
<para>
|
|
This is a very important step. xdb_file,
|
|
the default xdb that comes with jabberd,
|
|
is limited to open file descriptors too.
|
|
You can play the same shell games that we
|
|
are going to play with jadc2s later, but
|
|
if you want a server that can handle
|
|
millions or users, then you need something
|
|
other than xdb_file. Enter xdb_sql, which
|
|
only uses one file descriptor to connect
|
|
to the mysql server.
|
|
</para>
|
|
</note>
|
|
<para>
|
|
For more information on how to configure
|
|
xdb_sql, please see the README that is
|
|
distributed in the release.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Build jadc2s and distribute the binaries
|
|
to the boxes where they will run.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Setup the main jabberd to accept the jadc2s
|
|
component connections. In the jabber.xml
|
|
config file, add the following XML:
|
|
</para>
|
|
<screen>
|
|
<service id="jadc2s-1">
|
|
<accept>
|
|
<ip/>
|
|
<port>5111</port>
|
|
<secret>secret</secret>
|
|
</accept>
|
|
</service>
|
|
|
|
<service id="jadc2s-2">
|
|
<accept>
|
|
<ip/>
|
|
<port>5112</port>
|
|
<secret>secret</secret>
|
|
</accept>
|
|
</service>
|
|
</screen>
|
|
<para>
|
|
Now you can run the main jabberd and get it
|
|
listening for the jadc2s to connect to
|
|
it.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Configure the jadc2s.xml on each box to
|
|
connect to the SM, where to listen, etc...
|
|
</para>
|
|
<screen>
|
|
<!-- session manager configuration -->
|
|
<sm>
|
|
<!-- host and port of the SM -->
|
|
<host>localhost</host>
|
|
<port>5111</port>
|
|
<!-- shared secret, for authenticating us -->
|
|
<secret>secret</secret>
|
|
|
|
<!-- our ID, for authenticating us to the sm -->
|
|
<id>jadc2s</id>
|
|
|
|
<!-- how many times to try to connect to the sm (default: 5) -->
|
|
<retries>5</retries>
|
|
</sm>
|
|
|
|
<!-- local network settings -->
|
|
<local>
|
|
<!-- who we identify ourselves as. This should correspond to the -->
|
|
<!-- ID (host) that the session manager thinks it is. -->
|
|
<id>localhost</id>
|
|
|
|
<!-- IP address to bind to (default: 0.0.0.0) -->
|
|
<!-- <ip>0.0.0.0</ip> -->
|
|
|
|
<!-- port to bind to (default: 5222) -->
|
|
<port>5222</port>
|
|
|
|
<!-- SSL configuration -->
|
|
<!-- Specify the port to listen on, and the key file to use for -->
|
|
<!-- the certificate. -->
|
|
<!-- <port/> (default: 5223) -->
|
|
<!-- <pemfile/> (default: ./server.pem) -->
|
|
<!--
|
|
<ssl>
|
|
<port>5223</port>
|
|
<pemfile>./server.pem</pemfile>
|
|
</ssl>
|
|
-->
|
|
</local>
|
|
</screen>
|
|
<para>
|
|
For more information on how to configure
|
|
jadc2s, please see the README in the jadc2s
|
|
source directory.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Open a shell where you can change file
|
|
system parameters (root usually) and execute
|
|
the following command:
|
|
</para>
|
|
<informalexample>
|
|
<screen format="linespecific">
|
|
<prompt moreinfo="none">bash$</prompt> <command>echo "24000" > /proc/sys/fs/file-max</command>
|
|
</screen>
|
|
</informalexample>
|
|
<para>
|
|
This bumps the upper bound on the
|
|
number of allowed file descriptors
|
|
that can be open at one time.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Set the limit for the shell you are in
|
|
to use more than the default 1024 file
|
|
descriptors.
|
|
</para>
|
|
<informalexample>
|
|
<screen format="linespecific">
|
|
<prompt moreinfo="none">bash$</prompt> <command>ulimit -n 11000</command>
|
|
</screen>
|
|
</informalexample>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Tell jadc2s how many file descriptors it
|
|
is allowed to use:
|
|
</para>
|
|
<screen>
|
|
<!-- maximum number of file descriptors. Should be a prime -->
|
|
<!-- number. At least four will be used by jadc2s itself, -->
|
|
<!-- usually around six or seven (default: 1023) -->
|
|
<!-- For a list of prime numbers: -->
|
|
<!-- http://www.utm.edu/research/primes/lists/small/10000.txt -->
|
|
<max_fds>10007</max_fds>
|
|
</screen>
|
|
<para>
|
|
It is important that the number you pick is
|
|
a prime number. To make it easy to find the
|
|
prime you want you can visit this page
|
|
<ulink url='http://www.utm.edu/research/primes/lists/small/10000.txt'>http://www.utm.edu/research/primes/lists/small/10000.txt</ulink>.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
All that's left is to run the server, and the
|
|
jadc2s processes. Everything should work fine.
|
|
If it doesn't, then PLEASE tell me at
|
|
<email>reatmon@jabber.org</email> so that I can
|
|
fix this document.
|
|
</para>
|
|
</listitem>
|
|
</orderedlist>
|
|
</section>
|
|
|
|
</section>
|
|
|
|
<section id='redirection'>
|
|
<title>Connection Redirection</title>
|
|
|
|
<section id='redirectiondream'>
|
|
<title>The Dream</title>
|
|
<para>
|
|
Now that we have the c2s processes distributed
|
|
across machines, we need to get them to work
|
|
together. We need two things, the ability to
|
|
communicate how many connections a c2s has to
|
|
the others, and the ability to know who the
|
|
others are.
|
|
</para>
|
|
<para>
|
|
[22:21] Jer: well, the "protocol" could just be
|
|
an auth error of 3xx for redirect w/ the ip/port
|
|
to connect to
|
|
</para>
|
|
<para>
|
|
Maybe we need to talk about the router layer sooner...
|
|
</para>
|
|
<para>Pros:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>Load balances nicely.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>Cons:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
Requires client support for connection
|
|
redirection. This is not hard to do,
|
|
it is just not part of any client right
|
|
now.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
</section>
|
|
|
|
<section id='routerlayer'>
|
|
<title>Router Layer</title>
|
|
|
|
<section id='routerlayerdream'>
|
|
<title>The Dream</title>
|
|
<para>
|
|
</para>
|
|
<para>Pros:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>Cons:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
</section>
|
|
|
|
<section id='jsmdb'>
|
|
<title>Moving JSM State data into a DB</title>
|
|
|
|
<section id='jsmdbdream'>
|
|
<title>The Dream</title>
|
|
<para>
|
|
</para>
|
|
<para>Pros:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>Cons:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
</section>
|
|
|
|
<section id='multijsm'>
|
|
<title>Multiple JSMs</title>
|
|
|
|
<section id='multijsmdream'>
|
|
<title>The Dream</title>
|
|
<para>
|
|
</para>
|
|
<para>Pros:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>Cons:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
</section>
|
|
|
|
<section id='jsmcomm'>
|
|
<title>JSM Communications</title>
|
|
|
|
<section id='jsmcommdream'>
|
|
<title>The Dream</title>
|
|
<para>
|
|
</para>
|
|
<para>Pros:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>Cons:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</section>
|
|
</section>
|
|
|
|
</chapter>
|
|
|
|
</book>
|
|
|