Contents:

Abstract

Introduction

A Message Transfer Service

The Protocol

Sender Oriented File Transfer Service

Conclusion

Acknowledgements

References


The IPSendfile Project

R.J.Kummerfeld
P. Lauder
Department of Computer Science
University of Sydney

Presented at Networkshop92, Brisbane, Australia.

Abstract

This paper describes a message oriented file transfer service for the Internet. Conventional methods for sending a data file to another user involve the use of mail or ftp. Using mail involves splitting and encoding files while use of ftp is inconvenient and insecure. We have developed a protocol for the transport of messages consisting of an arbitrary number of 8 bit bytes that is used as the basis for a sender intitiated file transfer service.

Introduction

The internet is used to transport data from machine to machine, user to user. A very common use is for one user to send a file to another user. Unfortunately, and this is very surprising, the Internet family of services does not include a convenient method for doing this.

The most natural approach would be simply to ask, using the user interface, for the file to be sent to a destination user address consisting of the username and domain. This would treat the file as a message containing arbitrary binary data of any length and deliver the file to some part of the user file space.

The first candidate for providing this type of service is the File Transfer Protocol or FTP [4]. FTP is an interactive service that requires an end-to-end connection with the destination host. The source user `logs in' with a username and password and then instructs the FTP program to move a file from their file space to the directory of the destination username.

The first problem with this is that the service is interactive and requires an end-to-end connection. Most users would rather send the file and get on with their work while the file is delivered to the destination. If there is a network problem, the file should be queued, and delivered when the problem is resolved. With FTP the user has to try again later manually.

The second problem is that the user must have a username and password on the destination machine. This is either inconvenient, in that the user then must have an account on every machine that a file is to be sent to, or is insecure if the destination user is to give out their username and password for the transfer.

There is a convention adopted by most systems that the user name `anonymous' can be used by anyone to retrieve files. This has been extended to the case of sending files by providing a writable directory, often called `incoming' (or was that `in.coming'?), where files can be left for a destination user. Once a file has been deposited in `incoming' you then have to send email to the destination user telling them there is a file waiting. This has the serious drawback that while the file is in `incoming' it can be viewed or deleted by other users.

FTP doesn't seem to be the answer to the file sending problem.

The messaging service that is almost universal on the Internet is electronic mail. It consists of a combination of SMTP [3] for message transport and RFC822 [1] for message format. Unfortunately, this service is strongly oriented towards reletively short electronic mail messages containing 7-bit US-ASCII characters and so is unsuitable as it stands for transporting binary files of any length. The RFC822 message format standard has recently been extended (MIME [5]) to handle arbitrary message types in different encodings, including arbitrary binary. However, this is an unnatural layering with a binary transport protocol (TCP) carrying a 7-bit US-ASCII electronic mail service which is being used to carry binary messages.

It is far more natural to implement a messaging service that carries arbitrary length binary messages and use it to carry higher level message services such as a sender oriented file transfer service.

The `Sydney Unix Network' software and its commercial successor `MHSnet' that formed the basis for the ACSnet network does have a suitable architecture. It also has a sender initiated file transfer system. To send a file to another person a command like:

netfile bob@cs.su.oz.au myfile

is used. The file is transported across the network by the MHSnet software and kept in a holding area on the destination machine. A message is then sent (via email) to the destination user notifying them that a file has arrived. The destination user then invokes the command:

netget

to retrieve the file.

MHSnet provides many more facilities than are required to build an Internet message service. It has a complete routing table management system, support for any virtual circuit with a range of delay and error characteristics and a session layer protocol for establishing connections. It is also a proprietary package and is not widely available on the Internet.

To provide a sender initiated file transfer service for the internet it was decided to build a basic message transfer service.

A Message Transfer Service

The Message Transfer Service carries messages across a TCP connection. Messages may contain an arbitrary number of 8-bit bytes and no interpretation is made of the message by the Message Transfer Service. Carried with each message are the addresses of the source and destination users and a message type and associated parameters. The Message Transfer Service will process the received message in a way appropriate to the message type. This is expected to be implemented by the Message Transfer Service passing the message to another process.

The Message Transfer Service has the ability to start a transfer at an arbitrary point in the message. This would normally occur when a connection has been re-established after an unexpected disconnect and is in response to a receiver asking that the message transfer begin at a nominated point.

The service consists of three programs:

sendmsg
places message on outgoing queue
sender
processes outgoing queue, sends messages
tcpsmd
receives messages, invokes receiving program for message type

A user invokes sendmsg with a destination user address, a message type and associated parameters and the data to send. The message is placed on a queue which is periodically scanned by sender which attempts to open a connection to tcpsmd on the destination machine and then, using the protocol described below, transfers the message. At the destination machine, tcpsmd receives the file and then invokes a program to process the particular message type involved.

The Protocol

The message transfer protocol is designed to be extremely simple and easy to implement. It may be possible to implement this service using a language like Perl or with the Bourne shell with a small number of support programs. The description of the protocol that follows is informal and designed to convey the flavour rather than the exact detail. For a more detailed description see the draft RFC.

A message transfer is initiated by a sender opening a full duplex TCP connection [2] to a receiver. All commands are transmitted as 7-bit US-ASCII characters in 8-bit bytes with the high-order bit set to zero. Each command is terminated by an ASCII newline character (012). All data is transmitted as 8-bit bytes.

The receiver begins the session by sending an ``ACCEPT'' string with zero or more options.

The sender then sends the sequence:

ID unique-id
TYPE name
RECEIVER local-part [@ domain ]
SENDER local-part
[ FLAG flag ]
[ PARAMETER param ]
[ TIMEOUT number ]
SIZE number

Items in square brackets are optional.

The unique identifier is a 14 character string of printable characters. The `name' is a message type, in the case of a file transfer it is ``file''. The `param' and `flag' are character strings consisting of fields separated by spaces.

The receiver responds by sending either:

START address

or

NAK reason

The `address' is a number giving the point in the file that the transmission begins. This would normally be zero but in that case of a restart after a connection is broken would be the address of the last received data. If the receiver cannot continue with the transfer, it sends NAK and a `reason' string.

If the receiver has responded with START, the sender continues with the line:

DATA start

In this command, `start' refers to the start address chosen by the sender which must be less than of equal to that chosen by the receiver.

This is followed by the data, exactly size-start bytes, and then

DEND crc

The `crc' field is a 32 bit CRC, calculated according to the Ethernet polynomial, over the size-start data bytes.

This transfer is now concluded by the receiver responding with either

ACK or NAK reason

At this point the receiver can close the connection to indicate it is unwilling to accept any more messages or if the connection remains open the process can repeat with the sender sending the initial sequence.

Sender Oriented File Transfer Service

Using the message transfer service we can easily provide a sender oriented file transfer service.

This consists of three programs:

sendfile
send a file to a destination user
filer
process a file message when received
getfile
retrieve a received file

The user invokes sendfile with the destination user address and the name of the file or files to be sent. This uses the sendmsg function to send a message of type file containing the files to the destination user. At the destination machine, TCPsmd receives the message and invokes filer to process it. Filer moves the file into a holding directory and notifies the destination user (via email) that a file has arrived. At some later time the destination user invokes getfile to retrieve the file.

Conclusion

A sender initiated file transfer system has been developed based on a more general message transfer system.

The software is freely available via ftp from ftp.cs.usyd.edu.au in the TCPmsg directory. It is designed to be portable across all Unix and Unix-like systems.

Acknowledgements

The authors wish to thank Chris Rusbridge of the University of South Australia for suggesting that the Internet could benefit from a sendfile facility similar to that available on ACSnet.

We also wish to thank the AARNet board for providing a grant to allow the work to be carried out.

References

[1] RFC822 - D.H.Crocker. Standard for the Format of ARPA Internet Text Messages.

[2] RFC793 - J. Postel (ed.). Transmission Control Protocol - DARPA Internet Program Protocol Specification.

[3] RFC821 - J. Postel, Simple Mail Transfer Protocol.

[4] RFC959 - J. Postel, J. Reynolds. File Transfer Protocol.

[5] RFC1341 - N.Borenstein, N.Freed, MIME (Multipurpose Internet Mail Extension).