First checkin

This commit is contained in:
reatmon 2002-08-03 22:46:55 +00:00
parent 0680253f86
commit befd17a9d8
2 changed files with 478 additions and 0 deletions

View File

@ -0,0 +1,3 @@
*.html
build
env_setup

View File

@ -0,0 +1,475 @@
<?xml version='1.0' encoding='ISO-8859-1'?>
<!DOCTYPE book PUBLIC '-//OASIS//DTD DocBook XML V4.1.2//EN'
"/usr/share/sgml/docbook/dtd/xml/4.1.2/docbookx.dtd" [
]>
<book id="index">
<bookinfo>
<title>Jabber Server Farming How-To</title>
<pubdate>2002-06-06</pubdate>
<author>
<firstname>Ryan</firstname>
<surname>Eatmon</surname>
<affiliation>
<orgname>Jabber.org</orgname>
<address><email>reatmon@jabber.org</email></address>
</affiliation>
</author>
<abstract>
<para>Provide Jabber admins a look into the present and future of Jabber Server Farming.</para>
</abstract>
<revhistory>
<revision>
<revnumber>0.1</revnumber>
<date>2002-06-06</date>
<authorinitials>rwe</authorinitials>
<revremark>First draft.</revremark>
</revision>
</revhistory>
</bookinfo>
<chapter id="aboutthishowto">
<title>About this How-To</title>
<section id='purpose'>
<title>Purpose / Scope</title>
<para>
This document was started on June 06, 2002 by Ryan Eatmon to
explain how to set up a Jabber Server Farm using the Jabber.org
Jabber Server (<ulink url='http://jabberd.jabberstudio.org'>http://jabberd.jabberstudio.org</ulink>).
</para>
<para>
This document is broken into two main sections. What is, and
what will be. As of the writing of this document the Jabber
Server does not have all of the pieces that are required to
do full farming. There are steps that can be taken to get
partial farming, and those will be covered. There are also
steps that we are going to take, and those are covered to so
that others can contribute and comment.
</para>
</section>
<section id='aboutjabber'>
<title>About Jabber</title>
<para>
Jabber is an Open Source Instant Messaging System that uses
XML as its base protcol. I could spend an entire How-To
describing Jabber, so I am just going to point you to the
Jabber.org web site. (<ulink url='http://www.jabber.org'>http://www.jabber.org</ulink>).
</para>
</section>
<section id="feedback">
<title>Feedback</title>
<para>
Comments on this How-To may be directed to the author
(<email>reatmon@jabber.org</email>).
</para>
</section>
<section id="copyrights">
<title>Copyrights and Trademarks</title>
<para>Copyright 2002 Ryan Eatmon</para>
<para>
Permission is granted to copy, distribute and/or modify
this document under the terms of the GNU Free Documentation
License, Version 1.1 or any later version published by
the Free Software Foundation; with no Invariant Sections,
with no Front-Cover Texts, and with no Back-Cover Texts.
A copy of the license is included in the appendix entitled
"GNU Free Documentation License".
</para>
<para>Jabber&#153; is a Trademark of Jabber, Inc.</para>
</section>
<section id="acknowledgements">
<title>Acknowledgments and Thanks</title>
<para>
Thanks to everyone who gave me hints and tips on tuning
Linux and structuring of the farm. This includes Thomas
Muldowney, Jeremie Miller, and other members of the
<email>jabberd@jabberstudio.org</email> list.
</para>
</section>
</chapter>
<chapter id='introduction'>
<title>Introduction to Jabberd and Farming</title>
<section id='jabberserver'>
<title>Jabberd - The Jabber.org Jabber Server</title>
<para>
The Jabberd server is a reference implementation for
the Jabber protocol as defined by the Jabber Software
Foundation. It is written in C and is covered by both
the Jabber Open Source License(JOSL) and the GNU Public
License(GPL). The Jabberd project has its own
documentation. (<ulink url='http://jabberd.jabberstudio.org/howto.html'>http://jabberd.jabberstudio.org/howto.html</ulink>).
</para>
</section>
<section id='farming'>
<title>Farming</title>
<para>
This is an interesting topic. When you think of a
server you tend to think of a single machine running
a single program. Farming is the idea of multiple machines
running multiple programs that all act together to
appear to be one machine/program.
</para>
<para>
Why farming? Two reasons with equal importance. Scalability
and Reliability.
</para>
<para>
Scalability is the idea of allowing something to handle
many many more connections/transactions at the same time.
A typcial web server can handle say 1000 connections a
second, but 10 web servers acting together can provide
10,000 connections per second.
</para>
<para>
Reliability is the idea that even if a piece of the system
goes down, the system does not. There is no single point
of failure, and there is built in redundancy to ensure
that something is there to pick up the slack if something
goes down. In the web server example, if one web server
goes down you could not send any connections to it until
it is back up, thus ensuring that your web site is always
up.
</para>
<para>
Both of these ideas are crucial to a successful Farming
strategy. Each wants to push the design into a different
direction, but they can be reconciled into a single
design that can scale and is reliable at the same time.
</para>
</section>
</chapter>
<chapter id='implementation'>
<title>Implementing the Farm</title>
<section id='c2sfarm'>
<title>Client-to-Server Farming w/ Round Robin DNS</title>
<section id='c2sfarmdream'>
<title>The Dream</title>
<para>
The first thing that we can do is to split out
the c2s from the server and make it a seperate
process/component. Once that is done, we can
replicate the c2s processes over multiple
machines. Each one would connect back to the
main server running the JSM and everything
should work just fine.
</para>
<para>Pros:</para>
<itemizedlist>
<listitem>
<para>Easy to setup right now.</para>
</listitem>
</itemizedlist>
<para>Cons:</para>
<itemizedlist>
<listitem>
<para>
Round Robin will not provide true load
balancing since there is no mechanism in
place to check how many connections a
server has when it forwards a new connection
to it the next time it comes up in the
Round Robin.
</para>
</listitem>
</itemizedlist>
</section>
<section id='c2sfarmreality'>
<title>The Reality</title>
<para>
This is doable right now. The 1.5 version of
jabberd has sought to break all of the pieces
out into seperate processes. This was not
done for farming specifically, but we will not
complain since the jadc2s component can handle
upwards of 10k users. (The 1.4.x series c2s
could only handle ~1024).
</para>
<para>
Currently there is a forked version of jadc2s
that works with the 1.4.2 server. It is located
in the jabberd14 CVS repository on JabberStudio.
The following example is running with two jadc2s
boxes, and one central jabberd box. To set this
up do the following:
</para>
<orderedlist>
<listitem>
<para>
Get all of the
<ulink url='http://jabberd.jabberstudio.org/'>source code for jabberd14</ulink>.
Build the server, configure/make.
</para>
</listitem>
<listitem>
<para>
Get the
<ulink url='http://download.jabber.org/contrib/xdb_sql-1.2.tar.gz'>source</ulink>
for xdb_sql from
<ulink url='http://www.IDEALX.org'>IDEALX</ulink>
, build it, and setup the jabber.xml to use it.
</para>
<note>
<para>
This is a very important step. xdb_file,
the default xdb that comes with jabberd,
is limited to open file descriptors too.
You can play the same shell games that we
are going to play with jadc2s later, but
if you want a server that can handle
millions or users, then you need something
other than xdb_file. Enter xdb_sql, which
only uses one file descriptor to connect
to the mysql server.
</para>
</note>
<para>
XXX Add Link, or docs for doing this.
</para>
</listitem>
<listitem>
<para>
Build jadc2s and distribute the binaries
to the boxes where they will run.
</para>
</listitem>
<listitem>
<para>
Setup the main jabberd to accept the jadc2s
component connections. In the jabber.xml
config file, add the following XML:
</para>
<screen>
&lt;service id="jadc2s-1"&gt;
&lt;accept&gt;
&lt;ip/&gt;
&lt;port&gt;5111&lt;/port&gt;
&lt;secret&gt;secret&lt;/secret&gt;
&lt;/accept&gt;
&lt;/service&gt;
&lt;service id="jadc2s-2"&gt;
&lt;accept&gt;
&lt;ip/&gt;
&lt;port&gt;5112&lt;/port&gt;
&lt;secret&gt;secret&lt;/secret&gt;
&lt;/accept&gt;
&lt;/service&gt;
</screen>
<para>
Now you can run the main jabberd and get it
listening for the jadc2s to connect to
it.
</para>
</listitem>
<listitem>
<para>
Configure the jadc2s.xml on each box to
connect to the SM, where to listen, etc...
XXX More info here.
</para>
</listitem>
<listitem>
<para>
Open a shell where you can change file
system parameters (root usually) and execute
the following command:
</para>
<informalexample>
<screen format="linespecific">
<prompt moreinfo="none">bash$</prompt> <command>echo "24000" > /proc/sys/fs/file-max</command>
</screen>
</informalexample>
<para>
This bumps the upper bound on the
number of allowed file descriptors
that can be open at one time.
</para>
</listitem>
<listitem>
<para>
Set the limit for the shell you are in
to use more than the default 1024 file
descriptors.
</para>
<informalexample>
<screen format="linespecific">
<prompt moreinfo="none">bash$</prompt> <command>ulimit -n 11000</command>
</screen>
</informalexample>
</listitem>
<listitem>
<para>
Tell jadc2s how many file descriptors it
is allowed to use:
</para>
<para>XXX config example here</para>
<para>
It is important that the number you pick is
a prime number. To make it easy to find the
prime you want you can visit this page
<ulink url='http://www.utm.edu/research/primes/lists/small/10000.txt'>http://www.utm.edu/research/primes/lists/small/10000.txt</ulink>.
</para>
</listitem>
</orderedlist>
</section>
</section>
<section id='redirection'>
<title>Connection Redirection</title>
<section id='redirectiondream'>
<title>The Dream</title>
<para>
Now that we have the c2s processes distributed
across machines, we need to get them to work
together. We need two things, the ability to
communicate how many connections a c2s has to
the others, and the ability to know who the
others are.
</para>
<para>
[22:21] Jer: well, the "protocol" could just be
an auth error of 3xx for redirect w/ the ip/port
to connect to
</para>
<para>
Maybe we need to talk about the router layer sooner...
</para>
<para>Pros:</para>
<itemizedlist>
<listitem>
<para>Load balances nicely.</para>
</listitem>
</itemizedlist>
<para>Cons:</para>
<itemizedlist>
<listitem>
<para>
Requires client support for connection
redirection. This is not hard to do,
it is just not part of any client right
now.
</para>
</listitem>
</itemizedlist>
</section>
</section>
<section id='routerlayer'>
<title>Router Layer</title>
<section id='routerlayerdream'>
<title>The Dream</title>
<para>
</para>
<para>Pros:</para>
<itemizedlist>
<listitem>
<para>
</para>
</listitem>
</itemizedlist>
<para>Cons:</para>
<itemizedlist>
<listitem>
<para>
</para>
</listitem>
</itemizedlist>
</section>
</section>
<section id='jsmdb'>
<title>Moving JSM State data into a DB</title>
<section id='jsmdbdream'>
<title>The Dream</title>
<para>
</para>
<para>Pros:</para>
<itemizedlist>
<listitem>
<para>
</para>
</listitem>
</itemizedlist>
<para>Cons:</para>
<itemizedlist>
<listitem>
<para>
</para>
</listitem>
</itemizedlist>
</section>
</section>
<section id='multijsm'>
<title>Multiple JSMs</title>
<section id='multijsmdream'>
<title>The Dream</title>
<para>
</para>
<para>Pros:</para>
<itemizedlist>
<listitem>
<para>
</para>
</listitem>
</itemizedlist>
<para>Cons:</para>
<itemizedlist>
<listitem>
<para>
</para>
</listitem>
</itemizedlist>
</section>
</section>
<section id='jsmcomm'>
<title>JSM Communications</title>
<section id='jsmcommdream'>
<title>The Dream</title>
<para>
</para>
<para>Pros:</para>
<itemizedlist>
<listitem>
<para>
</para>
</listitem>
</itemizedlist>
<para>Cons:</para>
<itemizedlist>
<listitem>
<para>
</para>
</listitem>
</itemizedlist>
</section>
</section>
</chapter>
</book>