197 lines
7.6 KiB
Plaintext
197 lines
7.6 KiB
Plaintext
|
Compressed TCP/IP-Sessions using SSH-like tools
|
|||
|
Sebastian Schreiber <Schreib@SySS.de>
|
|||
|
|
|||
|
2.2.2000
|
|||
|
|
|||
|
1. Introduction
|
|||
|
|
|||
|
In the past, we used to compress files in order to save disk space.
|
|||
|
Today, disk space is cheap - but bandwidth is limited. By compressing
|
|||
|
data streams, you achieve two goals:
|
|||
|
|
|||
|
1) You save bandwidth/transfered volume (that is important if you have
|
|||
|
to pay for traffic or if your network is loaded.).
|
|||
|
|
|||
|
2) Speeding up low-bandwidth connections (Modem, GSM, ISDN).
|
|||
|
|
|||
|
This HowTo explains how to save both bandwith and connection time by
|
|||
|
using tools like SSH1, SSH2, OpenSSH or LSH.
|
|||
|
|
|||
|
2. Compressing HTTP/FTP,...
|
|||
|
|
|||
|
My office is connected with a 64KBit ISDN line to the internet, so the
|
|||
|
maximum transfer rate is about 7K/s. You can speed up the connection
|
|||
|
by compressing it: when I download files, Netscape shows up a transfer
|
|||
|
rate of up to 40K/s (Logfiles are compressable by factor 15). SSH is a
|
|||
|
tool that is mainly designed to build up secure connections over
|
|||
|
unsecured networks. Further more, SSH is able to compress connections
|
|||
|
and to do port forwarding (like rinetd or redir). So it is the
|
|||
|
appropriate tool to compress any simple TCP/IP connection. "Simple"
|
|||
|
means, that only one TCP-connection is opened. An FTP-connections or
|
|||
|
the connection between M$-Outlook and MS-Exchange are not simple as
|
|||
|
several connections are established. SSH uses the LempleZiv (LZ77)
|
|||
|
compression algorithm - so you will achieve the same high compression
|
|||
|
rate as winzip/pkzip. In order to compress all HTTP-connections from
|
|||
|
my intranet to the internet, I just have to execute one command on my
|
|||
|
dial-in machine:
|
|||
|
|
|||
|
ssh -l <login ID> <hostname> -C -L8080:<proxy_at_ISP>:80 -f sleep
|
|||
|
10000
|
|||
|
|
|||
|
<hostname> = host that is located at my ISP. SSH-access is required.
|
|||
|
|
|||
|
<login ID> = my login-ID on <hostname>
|
|||
|
|
|||
|
<proxy_at_ISP> =the web proxy of my ISP
|
|||
|
|
|||
|
My browser is configured to use localhost:8080 as proxy. My laptop
|
|||
|
connects to the same socket. The connection is compressed and
|
|||
|
forwarded to the real proxy by SSH. The infrastructure looks like:
|
|||
|
|
|||
|
|
|||
|
|
|||
|
64KBit ISDN
|
|||
|
My PC--------------------------------A PC (Unix/Linux/Win-NT) at my ISP
|
|||
|
SSH-Client compressed SSH-Server, Port 22
|
|||
|
Port 8080 |
|
|||
|
| |
|
|||
|
| |
|
|||
|
| |
|
|||
|
|10MBit Ethernet |100MBit
|
|||
|
|not compressed |not compressed
|
|||
|
| |
|
|||
|
| |
|
|||
|
My second PC ISP's WWW-proxy
|
|||
|
with Netscape,... Port 80
|
|||
|
(Laptop)
|
|||
|
|
|||
|
|
|||
|
|
|||
|
3. Compressing Email
|
|||
|
|
|||
|
3.1. Incoming Emails (POP3, IMAP4)
|
|||
|
|
|||
|
Most people fetch their email from the mailserver via POP3. POP3 is a
|
|||
|
protocol with many disadvantages:
|
|||
|
|
|||
|
|
|||
|
1. POP3 transfers password in clear text. (There are SSL-
|
|||
|
implementations of POP/IMAP and a challenge/response
|
|||
|
authentication, defined in RFC-2095/2195).
|
|||
|
|
|||
|
2. POP3 causes much protocol overhead: first the client requests a
|
|||
|
message than the server sends the message. After that the client
|
|||
|
requests the transferred article to be deleted. The server confirms
|
|||
|
the deletion. After that the server is ready for the next
|
|||
|
transaction. So 4 transactions are needed for each email.
|
|||
|
|
|||
|
3. POP3 transfers the mails without compression although email is
|
|||
|
highly compressible (factor=3.5).
|
|||
|
|
|||
|
You could compress POP3 by forwarding localhost:110 through a
|
|||
|
compressed connection to your ISP's POP3-socket. After that you have
|
|||
|
to tell your mail client to connect to localhost:110 in order to
|
|||
|
download mail. That secures and speeds up the connection -- but the
|
|||
|
download time still suffers from the POP3-inherent protocol overhead.
|
|||
|
|
|||
|
|
|||
|
|
|||
|
It makes sense to substitute POP3 by a more efficient protocol. The
|
|||
|
idea is to download the entire mailbox at once without generating
|
|||
|
protocol overhead. Furthermore it makes sense to compress the
|
|||
|
connections. The appropriate tool which offers both features is SCP.
|
|||
|
You can download your mail-file like this:
|
|||
|
|
|||
|
|
|||
|
|
|||
|
scp -C -l loginId:/var/spool/mail/loginid /tmp/newmail
|
|||
|
|
|||
|
|
|||
|
|
|||
|
But there is a problem: what happens if a new email arrives at the
|
|||
|
server during the download of your mailbox? The new mail would be
|
|||
|
lost. Therefore it makes more sense to use the following commands:
|
|||
|
|
|||
|
ssh -l loginid mailserver -f mv /var/spool/mail/loginid
|
|||
|
/tmp/loginid_fetchme
|
|||
|
scp -C -l loginid:/tmp/my_new_mail /tmp/loginid_fetchme
|
|||
|
|
|||
|
A move (mv) is a elementary operation, so you won't get into truble if
|
|||
|
you receive new mail during the execution of the comands. But if the
|
|||
|
mail server directories /tmp/ and /var/spool/mail are not on the same
|
|||
|
disc you might get problems. A solution is to create a lockfile on the
|
|||
|
server before you execute the mv: touch /var/spool/mail/loginid.lock.
|
|||
|
You should remove it, after that. A better solution is to move the
|
|||
|
file loginid in the same directory:
|
|||
|
|
|||
|
ssh -l loginid mailserver -f mv /var/spool/mail/loginid
|
|||
|
/var/spool/mail/loginid_fetchme
|
|||
|
|
|||
|
After that you can use formail instead of procmail in order to filter
|
|||
|
/tmp/newmail into the right folder(s): formail -s procmail <
|
|||
|
/tmp/newmail
|
|||
|
|
|||
|
3.2. Outgoing Email (SMTP)
|
|||
|
|
|||
|
You send email over compresses and encrypted SSH-connections, in order
|
|||
|
to:
|
|||
|
|
|||
|
|
|||
|
<20> Save network traffic
|
|||
|
|
|||
|
<20> Secure the connection (This does not make sense, if the mail is
|
|||
|
transported over untrusted networks, later.)
|
|||
|
|
|||
|
<20> Authenticate the sender. Many mail servers deny mail relaying in
|
|||
|
order to prevent abuse. If you send an email over an SSH-
|
|||
|
connection, the remote mail server (i.e. sendmail or MS-exchange)
|
|||
|
thinks to be connected, locally.
|
|||
|
|
|||
|
If you have SSH-access on the mail server, you need the following
|
|||
|
command:
|
|||
|
|
|||
|
ssh -C -l loginid mailserver -L2525:mailserver:25
|
|||
|
|
|||
|
If you don't have SSH-access on the mail server but to a server that
|
|||
|
is allowed to use your mail server as relay, the command is:
|
|||
|
|
|||
|
ssh -C -l loginid other_server -L2525:mailserver:25
|
|||
|
|
|||
|
After that you can configure your mail client (or mail server: see
|
|||
|
"smarthost") to send out mails to localhost port 2525.
|
|||
|
|
|||
|
4. Thoughts about performance.
|
|||
|
|
|||
|
Of course compression/encryption takes CPU time. It turned out that an
|
|||
|
old Pentium-133 is able to encrypt and compress about 1GB/hour --
|
|||
|
that's quite a lot. If you compile SSH with the option "--with-none"
|
|||
|
you can tell SSH to use no encryption. That saves a little
|
|||
|
performance. Here is a comprise between several download methods
|
|||
|
(during the test, a noncompressed 6MB-file was transfered from a
|
|||
|
133MHz-Pentium-1 to a 233MHz Pentium2 laptop over a 10MBit ethernet
|
|||
|
without other load).
|
|||
|
|
|||
|
|
|||
|
|
|||
|
+-------------------+--------+----------+-----------+----------------------+
|
|||
|
| | FTP |encrypted |compressed |compressed & encrypted|
|
|||
|
+-------------------+--------+----------+-----------+----------------------+
|
|||
|
+-------------------+--------+----------+-----------+----------------------+
|
|||
|
| Elapsed Time | |7.6s | 26s | 9s | 23s |
|
|||
|
+-------------------+--------+----------+-----------+----------------------+
|
|||
|
| Throughput | 790K/s | 232K/s | 320K/s | 264K/s |
|
|||
|
+-------------------+--------+----------+-----------+----------------------+
|
|||
|
|Compression Factor | 1 | 1 | 3.8 | 3.8 |
|
|||
|
+-------------------+--------+----------+-----------+----------------------+
|
|||
|
|
|||
|
|
|||
|
|
|||
|
5. Greetings
|
|||
|
|
|||
|
Thanks to Harald K<>nig <koenig@tat.physik.uni-tuebingen.de>, who used
|
|||
|
rcp in order to download complete mailboxes. The latest version of
|
|||
|
this howto is available on http://www.syss.de/howto.
|
|||
|
|
|||
|
|
|||
|
|