Return to the Cryptix Index. Headings below return to the above table of contents.
Please don't post large source to this group. Send it to the intended recipient directly. As the state of the lists is such that individuals are keeping their own archives, sticking source in doesn't help.
Ian has set up a wonderful archive . No more humungous mailboxes !
The team is a volunteer group. No funds are currently provided.
Support for team activities is on an ad-hoc basis.
The Cryptix Homepage is a stalled at the moment. The important
contents have been shifted to the distribution, and the Mirrors
perform the normal functions of the homepage. Much of the old
contents can be found in guide/cryptix
.
We have the three cryptix.* domains. The issue is where and how to site them. One view is:
Mailgroups.
Here is a good introduction to CVS. There are more resources listed below.
You will need both SSH (I use 1.2.20) and CVS (I use 1.9.10, it needs to be at least client-server enabled) installed on your system. It should be working.
In order to access the archive, create an SSH key:
$ ssh-keygen Initializing random number generator... Generating p: ........++ (distance 100) blah blah ... Enter file in which to save the key (/home/me/.ssh/identity): cryptix-cvs Enter passphrase: <<=== remember this bit Enter the same passphrase again: Your identification has been saved in cryptix-cvs. Your public key is: 1024 43 6425more numbers ...5623 me@crypto.hackers.com Your public key has been saved in cryptix-cvs.pub $
The public key is the long printed line of numbers terminating in your
email address. It is also in the file cryptix-cvs.pub
.
Send this to the
archive maintainer
for adding into the CVS authorized_keys
file.
To avoid Mallory joining the party, send it PGP encrypted and signed.
Please don't use this key for any other access, create other keys for other purposes. See below for how to set up the key as a different name and wrap a script around it.
export CVSROOT=:ext:cryptix@helplets.com:/home/cryptix/cvsroot export CVS_RSH="/usr/local/bin/ssh"in your .profile or equivalent. Notes:
/usr/bin/ssh
.
cvsroot
within the /home/cryptix
account.
cvs get READMEwhich will explain some archive details (not really written yet, still a test file). Once you get that file, you have proven access, so can go onto the next step, Usage, below.
cryptix-cvs
, above
#!/usr/local/bin/bash # set -x # uncomment for debugging ssh_gateway="ssh gateway.org" # set this to null if no gateway ssh_identity="-i .ssh/helplets" # use this key only for helplets # identity file is at the gate exec ${ssh_gateway} ssh ${ssh_identity} $*Then, I set the CVS_RSH (above) to call this program directly, and created the named key (above) on the firewall
gateway.org
(the files helplets*
are located in the .ssh
directory).
Once set up, when run, you should see something like
(with set -x uncommented):
me$ cvs get README
+ ssh_gateway=ssh gateway.org
+ ssh_identity=-i /home/me/.ssh/helplets
+ exec ssh gateway.org ssh -i /home/me/.ssh/helplets helplets.com -l cryptix cvs server
Host key not found from the list of known hosts.
Are you sure you want to continue connecting (yes/no)? yes
Host 'helplets.com' added to the list of known hosts.
cryptix's password:
Permission denied.
cvs [checkout aborted]: end of file from server (consult above messages if any)
At which point it is time to send the new key.
Note that to setup an ssh through a gateway,
you need account access to the gateway.
SSH currently doesn't support proxy access directly,
but it is being added.
Access - MS
Here are
Ian's
instructions for MS platforms based on his experiences
with NT. They should work for W95. Please send any experiences to
the list.
Software Needed
Download one of these.
-
Also try
Putty which is UK stuff. It doesn't do port forwarding though.
-
From NTK (except readers say Putty, above, is better):
>> TRACKING <<
look what the mouse dragged in
Until recently, if you wanted a Windows secure shell
connection to your Unix box, correct medicinal procedure was
to download the official Data Fellows software, use it until
the trial period expired, try to crack it, fail, then give
up and install Linux. Nowadays, even that doesn't work,
because those same fellows of Data are trying to encourage
everyone to upgrade to their new SSH2 protocol (special
powers include ultra-secure complete-incompatibility-with-
the-ssh-your-Unix-box-uses mode), and only the most
outrageous warez dudes can find their original, working,
software. Thus, the sudden renaissance of TERATERM PRO, an
ancient Windows terminal program that has the singular
advantage of having an ssh plug-in. Okay, it's not new,
exactly, but one day soon you'll need to find it, and
careful observers will note that no new software has been
invented since Netscape 1.1N. And even that was just
Hypercard with a glowing 'N', right?
http://www.zip.com.au/~roca/ttssh.html
- caution, Americans! RSA don't want our stinking foreign
code in your PCs
http://hp.vector.co.jp/authors/VA002416/teraterm.html
- Japan, too. Smacks of World Government to me.
-
Sascha put together this
one-stop-shopping
archive from these sources:
For cvs: I downloaded the normal cvs-1.9.16.tar.gz and fixed a bug on
Win32 when using an external RSH program. The source can be requested
from me. The patch (for cvs-1.9.10) is available on the same site
that has the ssh binaries.
For ssh: I downloaded it from
bmrc.berkeley.edu
but the ssh-keygen does not work.
There are other sites offering ssh for Win32:
therapy
and
martin
The version from bmrc.berkeley.edu worked best for me.
-
SSH-keygen
and
SSH
and
CVS
from Ian's homepage, or
-
the same components or the one-stop-shopper collection from
Set's homepage
but note that Sascha reports connect problems and assertion failures
with the
ssh.exe
there.
Creating a Key
Execute
SSH-keygen.exe
to create a key pair.
Mail the contents of the
public key file (by default,
identity.pub),
PGP encrypted and signed, to
archive maintainer
Once the archive maintainer has received your e-mail with your new SSH
public key, he will add it to his list of authorised
users and reply to your message. You should now be able
to run CVS correctly. Try executing
cvs get README
to check everything is working
correctly.
Environment
Set the following environment variables (using System
Properties Environment under NT, or by putting an entry
in your Win95 autoexec.bat):
-
CVS_RSH should be set to the full path to your SSH.EXE file
(including the filename itself). Make sure the path does
not include any directories with spaces in their names as
this throws CVS.
-
CVSROOT should be set to
:ext:cryptix@helplets.com:/home/cryptix/cvsroot
Making SSH jump through hoops
Just as with Unix, it is possible to make SSH do tricks.
In order to go through a proxy machine, assuming you cannot
access the archive directly, you can point CVS_RSH to this batch file:
@echo off
ssh.exe -l login_name gateway.machine.com ssh -i .ssh/cryptix-cvs %1 %2 %3 %4 %5 %6
In this case, the first SSH will jump on to the gateway machine,
and there, run another (Unix) SSH to connect to the repository.
Tips.
-
In general, test SSH first, then test CVS.
-
The gateway needs to have the key .ssh/cryptix-cvs
in existance first, test this first from the gateway itself.
-
Once the Gateway is working, test the batch script by
running it:
ssh.bat helplets.com pwd
-
On MS, the SSH command needs to specify the remote identity,
as there is no strong identity in an MS system.
-
The batch script needs to emulate SSH, with all keys set
(possibly by default), but without the final identity.
-
Then test the CVS by doing a
cvs get README
-
To debug the batch script, edit out the @echo off
line and look at the commands generated.
Some Notes
Some comments:
-
On W95/applets et al Raif said:
There's a jCVS: a Java front end to UNIX CVS. Have a look at
<http://www.ice.com/cvs/>.
I dld it just in case it'll do the job ;-) I guess
I'll know better once it's up and accessible.
- from mpp@mitekgold1.miteksys.com :
... we are using CVS with SSH. I think that this would work
well from Unix hosts, but from PC's there isn't a good SSH client available.
We have been using SSH from DataFellows and it has bugs that make it hard
to use, especially on Windows 95. On 95 we see problems with data corruption
which makes it really hard to use. On NT I think we only saw problems with
uploading data to the server and we haven't seen corruption for a long time.
There is also a public port of SSH 1.2.17 to Win 95 and we began to use
it, so far nobody is reporting problems. This public port, though has
a few limitations, one of each requires setting up a new SSH connection
for every CVS transaction and typing in the password for every CVS
transaction. Still, it's better than not working at all.
You can also try www.datafellow.com - they have the commercial version,
but it does not work from the command line and has bugs in port forwarding.
The public port is the other way round - it works from command line but
does not do port forwarding.
Usage
The entire hierarchy can be seen by browsing at
http://helplets.com/java/cryptix-cvs/.
Here is a quick list of CVS commands:
-
cvs get current
gets the whole tree into cryptix/....
It's big, and this will takes some time.
cvs get current/doc/AllNames.html
will get that single file. Sub-directories work as well.
Nota Bene.
CVS gets the named tree into your current directory, not
just the named file.
That is, this will create a directory
current/doc
and put
AllNames.html
into it. The get
command is relative
to the top of the tree, not to your current position
in your own tree. It catches me every time...
-
cvs update
in any directory within the hierarchy (including the top) will
bring down and patch in any newer versions. If there is
a conflict, it will tell you, and tell you how to deal with it.
-
cvs commit [files...]
will send your files in the current directory
up to the CVS archive for incorporation.
-
cvs add files [...]
will add new files from your current directory
into the archive. For importation of new trees,
look up import
instead.
Nota Bene for MS users.
CVS will automatically convert the files from DOS
format to Unix format. If you are adding a non-text
file such as a GIF, then add the -kb option
cvs add -kb files [...]
to tell
CVS to not add this file with all carriage returns
stripped out.
The above commands should get you going,
but be prepared to dive into the doco,
see the resources below, when you want something
beyond the basics.
CVS is a big complex system,
with many features and many traps for young players.
That's another way of saying I haven't mastered it yet...
Resources
This is a list of resources on CVS.
-
An
Introduction to CVS
-
USDA CVS site
has these resources:
- The
CVS Story Book is a FAQ including lots of links to resources
- An
archive of mail in months on main page
-
mail search of posts
-
A
Mini FAQ on the CVS mail group
-
CVS tutorial
-
Cyclic
are commercial supporters of CVS.
They have an
ftp repository
.
-
The old (out of date)
1995 FAQ
-
OpenBSD's
Anon CVS
update page
Posting Progress
Unless your task is completely obscure or silly, it is customary
to post intentions, progress and completion notes to the mailgroup.
This is so that others can help, flag problems, discuss issues and
methods, or consult with you if they need to get into that area.
Creating a New Repository
These notes are here because it's the logical place, rather than there
being a need to create repositories often.
First thing is to make sure SSH and CVS are installed, as above. If
these don't work, at least in client mode, then there is little hope
for making a repository.
Next, make an account.
Next, set up an SSH directory and file with the authorized_keys in it
of all the permitted accessors. An alternate way to do this is to
leave off this file and distribute the password.
Finally, run the following command:
cvs -d $PWD init
from within the account.
To access it, set up the CVS environment variables as described above.
There are more things you can do - set up chroot and only-cvs access,
and make web archives, but I don't know how to do that yet.
Work In Progress
Current Active Projects
These are the current active projects.
Who Project Description Status
Twente
SSH
V2. Twente Uni team in full is
Jeroen
Edwin,
Erwin
old.
Sascha
SSL V2
secure browser protocols, we need a snapshot release
2nd release ?
Jeroen
IJCE
IJCE to spec V1.2
WIP
Raif
AES
NIST page describes.
live
LOKI'97, Serpent, Pentagon,
kit published
Paulo
EC
ElGamal over Elliptic Curves, Square (with Raif)
Analysis
Ian (Quincy)
Enigma
secure mailer plugin for all mail handlers. Also looking at guerrilla groups and PFS
published in Cj2.2, bug fixing, also in ByProxy.
Edwin
PGP
produce complete compatible PGP in Java - JCE then OP.
WIP
Erwin
WebFunds
see webfunds.org.
works, distributes. Needs OpenPGP
Table 1: Current active projects
Wish List
We want everything ... well, almost. Here's a current list of
wants, there may be others:
-
Hardware.
-
Oxford Hardware Compilation Group.
-
Eracom
have a PCI board that David Hook is writing to.
-
nCipher
have a SCSI device that JCP are writing to.
-
Dr Dobbs article. Look at their
author's notes.
Also, there is the
QuickStart notes.
-
Mathematical algorithms. See for example algorithms in Chapters
2,
3,
4,
5,
14
in C from Pate Williams for the book
Handbook of Applied Cryptography, Menezes, et al.
-
DSA algorithm.
We need the JCA algorithms for the free Java
implementations that do not/can not use the JCA. So we probably
also need the DSA sig algorithm. What others are in the SUN
provider?
-
P1363 standard. This is a new standard for public key cryptography
that we want to be compatible with. Includes things like:
- Rabin-Williams
- elliptic curve schemes
- OAEP encryption padding (an improvement over PKCS#1)
- standard ASN.1 types for keys, etc.
Their home page is here
(username "P1363", password "MarsRoks").
-
A course! Yes, I'd like to do an online, interactive course
in applied cryptography. Something like:
- coupled with Cryptix code: exercises, projects
- written or guided by an academic name
- a series of modules:
number theory, cryptography, practical, protocols
- along the style of the OU
- assume programming and some mathematical ability.
- exams? a certificate? a tutor mailgroup?
Here's a list of existing
meatspace courses.
-
PKI - either pick up SPKI or extend Open PGP.
-
PRNG - generation of random numbers. This area is
always controversial, so could do with some attention:
- analysis of random numbers generated by current methods
- simple test programs to analyse results
- new modules using different methods
- documentation of issues, work done, results of analysis
perhaps in a CAN.
-
IPSec - whilst C IP stacks aren't really our area, there
is some possibility that an all-Java kernel and IP stack
will find a place in things like the NC or smart cards.
-
The interesting and secure secret key algorithms in Schneier,
accompanied by test and examples. E.g.,
- GOST
- 3-WAY
- CA-1.1
- FEAL
- DESX (used by IPSec)
- RC5
- Snuffle :-)
- streams: WAKE ...
Actually, there are only a few of them left, and even the ones
mentioned above have variable records....
But, there is no new edition of
Schneier in the pipeline, so there is plenty of time.
-
Alternative PK algorithms and packet formats: DH, ...
-
Open PGP (based on pgp5.0) as it is being standardised by the
open-pgp
IETF group (see page for mailgroup).
A
working draft
for the standard is in progress.
Here is a page with all the
Open PGP Implementors.
-
Examples, documentation (CAN) for the use of Cryptix with Applets.
-
PKCS certificates.
-
Electronic Commerce protocols:
- SET
- ecash
- SOX
Or, do electronic commerce protocols belong elsewhere?
Opinions may differ.
Next Major Release
See the mirror site for the last major release.
The major release will be 3.0.4. It might include:
- snapshot of PGP
- native
Distribution
These are working notes. I have produced about 4 distributions
this way, so I'm getting closer to the truth.
We are still prototyping though.
Introduction
How CVS wants to see it
CVS works on the basis of tags which are an advancement
on other source code control system that rely on simply dates
and version or release numbers (CVS includes this as well).
Tags are alphanumeric labels for a bunch of source files.
A tag can only refer to a single version of a source file
at once, so when you specify a tag, you get a distribution
of the source according to how the tag is currently set.
There are several sorts of tags:
-
The import vendor tag is meant to describe who the source
came from. It is only set during an import, so if you are
doing an import, then you are the vendor. You should use
your normal user name as the vendor tag (stick to lowercase)
-
The import release tag varies with each different release.
This is your release number, you set it, so it might
increment each time you import a new version.
Note that this is probably numerical, and of the
form 1-2-3, which refers to your release 1.2.3.
-
Distribution tags are created when we export source to users.
In this case the team is the vendor (but the vendor tag as
mentioned above is only relevant for the importing end-user).
Distribution tags are used for exporting or getting source.
They can move around.
Here are some working distribution tags:
-
HEAD
is the default you receive when you do a new get or
an export. HEAD is the current best in the repository,
and doesn't generally need to be mentioned.
-
Cryptix_1-2-3
to indicate that this is the main team distribution
of source called version 1.2.3.
-
RPK_1-0-0
for specific distributions of source to a special customer.
-
If you ever want to see what tags are attached to a file that
you have out, run cvs status -lv which will show everything
that it can think of.
CVS wants us to create a distribution by tagging each file version
that is in the distribution. That's what we talk about further below.
Goals of the Distribution System
We have a number of primary goals in building a release system:
-
Simplicity of new releases. So that people can quickly get in
there and run the process. I don't want to do this forever, so my
system must be easily passed on to others.
-
Minimisation of later support hassles. This means that it
is impractical to just zip up the lot and send it to the users.
What is called for here is a testing phase within the
distribution process that runs all
unit tests
on the very distribution
itself, so that we know that all tests will work for users
(modulo new bugs).
-
Efficiency of new releases. It should be possible to bash one out
quickly, so that we can set up lots of new releases (c.f.
Cathedral
). This of course is only useful if each release works.
There's also a number of secondary goals - ones that I might sacrifice
just to get the ball rolling:
-
Production of documentation (i.e., javadoc).
This will very quickly become
of more importance, in order to reduce support and get users
using the software.
-
Allow users to incorporate the distribution and then to make
their own custom distributions.
-
Reproducibility of results. As we get more professional
at this, we will need to rebuild releases from time to
time. But, in my experience, this is generally avoidable,
and is not really of great benefit unless we are supporting
some particular client with special needs.
Overview of System
All the above considerations leads me to the following rough architecture:
From CVS itself, only do source releases. Then, within the
source release, provide the tools (such as the src/build/*.sh scripts)
to build the binary distributions.
In order to do the source distribution,
all source that has to go in there has to be tagged with
the distribution tag.
This is a manual task of some tedium (see util/dist_tag.sh
)
This makes sense in that any distribution requires a lot
of difficult questions as to what goes in and what is omitted.
Each decision is instantiated by applying the distribution
tag to each file, meaning that it is chosen for inclusion.
Nota bene that it is the
distribution maintainer's
job to do this, not the programmers'. The distribution will
have a whole bunch of test scripts and so forth that will
bounce if not done properly. When programmers wishes to
add something into the distribution, they should coordinate with
the maintainer on this.
Then, when files are tagged, do a new
cvs get -r Cryptix_1-2-3 current
The files in this tree will be sticky to 1.2.3, so the process
is one of testing, fixing files, changing the tags on fixed files,
and then updating this tree.
A script in the source can produce a source release
(see util/dist_source.sh
)
or the whole release
(see util/dist_all.sh
).
This latter script produces a source release,
unpacks it, and produces the binary from that.
A script in the CVS source should
produce a source release, which should itself be
capable of producing a binary distribution.
This binary should be produced from the source release,
and then extracted into a temporary directory.
There, run the tests. As problems arise, fix the files, and make
a new distribution. As fixes work, they need to be tagged
with the distribution tag.
When the entire cycle is error-free,
and the set of files (actually, revisions) to distribute is tagged,
throw away all the above, and do a new
cvs export -r Cryptix_1-2-3 current
And then run through the process again. The test results
duplicate the previous results (this hasn't been tried to date).
Then copy the
source zip and other (binary or docs) zips to the archive,
and announce the new distribution.
Tagging
The first step is to decide on the distribution tag.
For sake of argument, we assume here it is an increment on
the previous one, and that makes it Cryptix_1-2-3.
Then, in a hierarchy, go through the laborious process of
tagging each file and module with that tag. Technically this
is done with:
cvs tag Cryptix_1-2-3 [files...]
It's probably worth doing this at least once manually,
just to get the feel of the distribution.
Notes:
-
Name the files explicitly or get the default of the
current directory
(and it will be recursive unless you add the
-l
option).
-
If you want to start from the contents of the previous release,
use the
-r old_tag
option:
cvs tag -r Cryptix_1-2-2 Cryptix_1-2-3 [files...]
where 1.2.2 is the previous, and 1.2.3 the new.
-
There is a script that automates most of it:
util/dist_tag.sh util/man/cvs.man
where the file is a manifest specifying all files and
directories.
That manifest needs to be read carefully each time,
at least to set the release tag.
-
Later on, to retag a new (fixed) version of a file,
you will need to add the -F flag to force moving the tag
to the current version. I use the following in a .profile:
push()
{
cvs commit $*
cvs tag -F ${CRYPTIX_CURRENT} $*
}
export CRYPTIX_CURRENT=Cryptix_3-0-3
and then for each new file that is changed,
a simple push file.java
will commit and tag it.
(dist_all.sh
will do an update in the build tree.)
-
This tagging process can be done in any working directory
where you can get access to the files (versions) that you
want. It is not likely to be the same as the working
directory out of which you build the distribution because
in that hierarchy you can only see the already tagged versions.
For example, it might be your current programming directory.
Building the Distribution
An important first point is to decide what is in the distribution.
These are decided by the files util/man/source.man
and util/man/binary.man
. They should be adjusted
to include new source and its unit tests. Then they should
be retagged.
Once everything is tagged, extract a new CVS directory:
mkdir 345
cd 345
cvs get -r Cryptix_3-4-5 current
Then, the process is one of checking the contents, running the script:
cd current
util/dist_all.sh
checking again, and rerunning the script.
Once the distribution gets to the end,
and all tests pass, add the
-export
flag (which is only locally set up, this part will need rewriting)
to export the distributions up to the web server.
See the
util index
for more information on the scripts.
Some points to make:
-
dist_binary.sh
,
buildjar.sh
,
testjar.sh
,
have to be in the source distribution.
This might change.
-
testjar.sh makes scripts called auto/testjar* that
can be run to reproduce its test actions.
-
The scripts are big and hairy, and are only really suitable
for big distributions. For day-to-day compilation, the other
scripts in util are probably more suitable to local's needs.
-
This is probably a Unix only game. Although the scripts might
work on a real shell, I suspect the whole approach will need
re-thinking in order to work under Wintel.
-
util/dist_all.sh
builds all three current distributions.
It effectively does these three:
util/dist_src.sh
util/dist_bin.sh
util/buildjavadoc.sh
Each of the above can be done individually from any
valid source tree.
-
The distribution only does what it is told, and just tagging
a file for inclusion in the source release says nothing about
the eventual release status.
To adjust the release, change (and push as above)
the manifest files util/man/*.man
and all of the
manifests that are included within it. To get a new module
into the entire distribution, these should be checked:
util/man/source.man
util/man/binary.man
util/man/JAR.man
where is JAR is where the binary release goes.
-
For each new release, the current system requires that
each manifest version number be upped by one, and that
the version number in
util/dist_all.sh
be changed.
Adding Your Code
To get a new unit added into the distribution, mail the
distribution team.
asking for the unit to be added in, and include:
- name of unit test(s)
- location of files
That first requirement, the name of the unit test(s) is paramount.
With that, it is possible to determine the rest. Without that,
the addition will take twice or three times as long, as the test
will have to be written or cobbled together. This would generally
push the addition down in the priorities.
Some Other Comments
Build and distribution is a core skill for the team to develop,
and it is fair to say that we haven't really done much in this
area. To help us, here are some docs that comment on the
subject.
-
Programmers' Canvas talks about some principles of team
productions:
-
The Cathedral and the Bazaar
also talks about how Linux manages big builds
(needs a list of specific sections)
Programming Standards
Security
I suspect this is something of an open ended section.
Baltimore makes an interesting claim:
Note: Any serious Java implementation of cryptographic functionality
must include a mechanism for obfuscating sensitive information, such as
keys, in memory. This will inevitably slow down the operation of the
toolkit.
David:
This is a hard problem in Java (or any garbage-collected language).
Obfuscating the keys isn't enough, because there will still be any number
of copies of data derived from the keys in memory. That's especially true
for the public key algorithms using java.math.BigInteger, since there is
no way to wipe intermediate results; you just have to rely on them being
garbage collected.
I remember seeing a hack for Linux posted to a newsgroup
(comp.security.unix?) showing how to disable virtual memory for an
particular program - the example was PGP 2.6.x. It required installing
the program suid root, and patching it to make a kernel call
(mlockall(), IIRC) then drop privileges.
Installing the java interpreter suid root is a definite no-no for
obvious reasons, but maybe we could provide a native C wrapper for
Linux that would call it. The newsgroup discussion implied that it was
possible to do a similar thing for Windows NT (but not 95 :-( ).
Getting the Best out of Cryptix
The benefit of strong crypto is best achieved by a balance
of code, documentation and testing, as this allows the
best use of limited resources.
power(Cryptix) = constant * code * doco * tests
Cryptix is now an internationally-used strong crypto package.
Now that many people are using the product,
it is important that each distribution not generate work for
us by causing users to ask obvious questions, trip up
on silly bugs, or grumble about the product because there are
bits missing.
There are several aspects to the overall process of producing
quality code:
-
Write a unit test class. A
Unit Test
is a simple program
that just tests the basic functionality.
These are essential for distribution, see below.
-
Write great doco. See the Documentation
section below.
-
Write a more complex thrash test. Where there is some uncertainty
about the code, a thrash test can pump lots of data through it
and try and trip up problems based on the type of input.
It might:
- encrypt-decrypt-check from a source of random data
- compare results against a known good source by
taking two streams in, processing the first and
comparing with the second.
- work from a seed by doing a basic test, then
hashing the seed and repeating.
-
Write an example program. This would seek to do something
generically useful with the class:
- encrypt a file
- generate performance benchmarks
- examine the data generated by other tools
(e.g., pgp keyring revealer).
Don't forget to include a simple self test so that the
distribution can pick up silly errors.
Which of Grandma's eggs you wish to suck is dependent on the moment!
Unit Tests
Testing is no longer a nicety in Cryptix, it's a necessity.
Cryptix has grown to the point where it can no longer be tested
manually, by just running a bunch of programs and looking at the results.
During distribution, each part of Cryptix is tested (sometimes
multiple times) to prove that it is good.
The objective of the distribution testing is to eliminate
errors in the distribution phase. It cannot eliminate
coding errors, and it can't improve the basic product.
Without automated testing, the complexity and size of the
product is such that there will be sufficient errors in
every distribution such as to make it unusable.
So in a sense, the objective
can be read as re-making Cryptix usable for the users.
To meet the objective,
the distribution scripts needs to prove that each part of Cryptix
is present and working.
This is done by running the unit tests which are,
literally, tests of component units of the whole.
The main feature of these tests is that
they provide a positive indication that the code is present,
and that it is working.
A unit test should conform to the following:
-
Testing should be sufficient to conclude:
- Class files are present and available for linking.
Especially, watch out for inner classes that are
compiled into distinct
parent$inner.class
files.
- The basic mechanism of the code works.
- Major integration is working, e.g., cipher modes.
- The major API calls are working.
-
Algorithm tests - those of ciphers, digests and PKs - should
-
Test the basic cycle of operation against
a pre-calculated set. That is, it is insufficient
to simply encrypt-decrypt and check. A cipher should
compare the ciphertext against known good ciphertext.
-
Include all known conformance sets.
This allows the documentation to assert that
the implementation conforms to the stated standard,
without having to be worried about when and which
version was tested for this.
-
Testing is called by identifying the entry class.
This is normally the main of a custom test class,
for example
cryptix.test.TestLOKI91
,
but it doesn't have to be. If another class is used,
it cannot, however, be inaccessible (i.e., it
cannot be private).
-
The unit test is run without arguments. This is
more a saving for the distribution scripts that
cannot remember which arguments are relevant.
The same program can still be used, with arguments, to
achieve more comprehensive tests.
-
Positive indication is returned, being one of
the exit codes in
Table 2, below.
-
The unit test should take less than a minute,
and hopefully less than 10 seconds.
This is because the distribution process
runs dozens of tests, and if each are long
then the overall process becomes too long to debug.
-
The unit should not be circularly dependent
on any other, and especially, neither should a
unit test rely on modules that are not yet tested
themselves.
Hierarchical dependencies are a natural part of
good building block design, but when blocks start to
prop each other up, problems arise.
( Explanatory note.
The main example here is the current IJCE tests. The
test that is run, cryptix.test.TestIJCE is also
dependent on the Cryptix Provider being present.
This means that basic failures in the IJCE are not
picked up immediately, and that results returned
by this test must be further dissected into the
units involved. )
Once the unit test is ready, send the class name entry point to
the
distribution team.
In fact, sending in the test is the most important thing that should be
done, if the issue is getting the code into the distribution.
From the test, it is fairly easy to work out what
code is required. Without the test, the distributor
needs to look for the test, which is fraught with errors, or
to write a test, which is inefficient.
Native Testing
This has not yet been tried at the distribution level.
The testing of native code currently borrows
from the existing algorithm unit tests, which
leaves a number of shortfalls as implied below.
Unit testing of native code will need to ensure that
-
both implementations are tested
-
returns a positive indication that both are working
-
returns a failure otherwise (perhaps returning the exit code
4,
carry on,
if only one of the two is working)
-
can distinguish between
"native is supposed to be there but doesn't work" and
"native is not written and should not be there."
One possibility is to run each unit test twice,
once for native, once for java. This could be done
simply by including different entry points into the
tests:
java cryptix.test.TestLOKI91.native
java cryptix.test.TestLOKI91.java
Another possibility is that the unit tests
could be upgraded to cope.
This is not a big issue in advance, the most important
step is to get running native code on the distribution
platforms (Unix) so that the strategy can be worked out.
Exit Codes
A test should exit according to the conventions in Table 2, which are
interpreted by distribution scripts.
no failures failures
no successes
no tests = 3
bad = 1
successes
good = 0
carry on = 4
Table 2: Matrix of Test Exit Codes
The purpose of the mixed-results exit code 4 is to allow
the distribution scripts to carry on and test the entire
set of tests, without forcing a debugging phase for each
single bug found. It's a problem of efficiency, as normally
found in compilers' reporting of errors and so forth.
If the carry on result cannot be provided, then
bad indication of 1 should be returned.
In Java, to return the code, call System.exit(n)
where n
is the exit code desired.
Documentation
Any documentation and testing that you can supply will be
immensely appreciated by the users.
Documentation can defer questions being asked over and over
again on the mail groups. By making the job of the user
easier, we make the amount of support less and can spend
more time on new code.
There are several ways you can do this:
-
Write comments in the code that describe what's going on (not what
the code means, it can be assumed that the reader well-understands
the language).
-
Write a javadoc section at the top.
The Javadoc is the place for any references to standards
or papers from which the code is derived, and the source
of any test data for conformance purposes.
-
Tests make for good documentation, as discussed above.
For some people, tests are the only documentation,
after that it is time to read the source.
-
Write a
Cryptix Application Note to describe
some deeper use of the code. This is intended to explain or
document something that cannot be done in the code.
Everybody is encouraged to work together on updating the
documentation; please be careful about preserving other people's
literary achievements.
Applets
There are problems with some browsers (I.E. and HotJava) if using
Cryptix from applets. Some of the examples allow arbitrary files
to be read or written. To combat this:
-
Examples should not be included in the straight classes jar files.
They can happily go in a separate jar file.
-
Application classes should be non-public.
-
Application classes wherever possible
should use System.in and System.out for I/O (i.e. act as a filter)
rather than using FileInputStream, etc.
Historical note: there may be more problems using Cryptix from applets,
as Cryptix was never intended to be used by applets.
Whitespace
Our working white space convention is:
-
4 spaces should be used for each level of nesting,
-
comments on the left using spaces,
-
Tabs should not be used at all, as they suffer mangling when
they go through the various conversions.
-
Line endings in Unix format (LF only).
In the source release there
is a Java application called
util.FixLineEndings
that can be used to convert tabs to spaces,
and line endings to any format.
Make sure that the root of the source release is on the CLASSPATH,
then do
"java util.FixLineEndings"
to get a summary of options.
There are also shell scripts to convert the whole source tree at once.
For example:
sh util/unixify.sh
or
sh util/dosify.sh
CVS should probably deal with CRs. Iang's guess at this stage
(not having tried it) is to strip out CRs on input,
and add them in on output (only for Windows users).
This is a known problem for which there are known fixes.
Final Keyword
Fields should be final iff they are constants.
Classes should usually be non-final, unless they need to be final for
security (example: cryptix.provider.cipher.*).
Methods should usually be non-final, unless
-
they need to be final for security (this is quite rare).
-
they are implementing an existing API written by someone else that was
declared as final (example: some of java.security.*)
-
they are auto-generated (example: the Maker classes generated by JavaCC)
> It speeds things up, and may prevent security holes where
> people extend our classes deviously, but is it appropriate
> in a library?
Static and private methods are called using static binding, so they are
just as efficient as final methods (the 'final' qualifier is redundant for
these methods, although there are quite a few cases in Cryptix where it is
there anyway). In the cipher and md classes, almost all of the methods that
need to be called quickly are already private.
File Name Conventions
-
Stick to a 32 character maximum for filenames for now (and remember
that ".class" takes up 6 of those characters).
Many less popular platforms are still plagued by this limit.
Watch out for inner classes that produce longer compiled class names.
Politics, Culture, etc
This entire section is rantings by
iang
who is probably not of the same opinion as anyone else...
How is the team organised?
Organised? Not very.
In political terms, we are a socialist taskocracy or somesuch.
That is, power is shared amongst those who do the work.
Hacker power rules.
This effectively means that those who make the distributions say
what goes in them. Those who write a particular interface have
final say as to how it is done.
Those who write original code get to say how it is done. By doing it.
But they don't get any rights in later versions, as later editors have
the same rights.
The counterbalancing to this tremendous concentration of power in
the hands of the do-ers is the ability of the rest of the world to ignore,
rewrite or fix anything they do not like. Ignorance is the privilege of
the market, and a message for the developer; rewriting or fixing converts
critics into do-ers.
All this leads to several effects:
-
If you do not like the way things are going, do it your own way.
Including, making your own distribution.
-
The onus is on the original developer to document and explain what
was intended, so that gratuitous hacks by well-meaning disciples
are limited to the bits that really do need fixing.
There is an well-written paper by Eric S. Raymond called
The Cathedral and the Bazaar
that more or less agrees with this viewpoint.
(However, it more or less disagrees with the following
"by invitation" approach).
Another description is the chapter on
"The Rise of ``Worse is Better''"
in an article on Lisp history by Richard P. Gabriel.
But this probably goes in a general software engineering section.
Why is the team "by invitation?"
Here is a mostly unedited mail describing the viewpoint of
iang
:
I think there are a lot of issues, which add up in my experience to a
conclusion that I think the group will work better if it is managed
as an "invite-only" club.
I am not sure what all these things are, but I'll have a go at it:
Industrial Espionage: yes, that's a concern. Whilst pure competition
doesn't count, there is plenty that an aggressive producer of crypto
could do to the group to kill it if the name of the game was to cause
Cryptix to wither and die. If I was Baltimore, I could have a lot of
fun. Even if I was Sun, I would have a lot of ability to influence the
fate of the team if I could see everything that was going on in it.
Team Size: there is an issue here in managing the amount of activity.
Whilst it is good to have lots of code happening, there is lots of
support that needs doing. At the moment, things are quiet, but over
the period of the release, many man-weeks were consumed in support
activities (femme-weeks can be included at premium rate here :))
If the team gets much more productive (that is, releases start happening
more quickly), then there is going to be a need to boost up all the support
side. That means things like the CVS and automatic distribution scripts.
Now, these are not technically out of our reach, but someone has to do all
the work. We have talked about it in the past, but it is no exaggeration to
say that the non-coding aspects of Cryptix can be extremely costly as well
as boring.
Noise level: with a team of this nature, it is important to keep everything
focused. If we have non-coders, or inactive people, that are not focused on
writing code, then there is a temptation for them to get involved and waste
other people's time. It's bad enough when we disagree amongst ourselves,
but at least we are all relatively quick to work out when the code is
suffering and make some sort of judgement as to whether a discussion
is still valuable.
So my current viewpoint is to keep my eye out for people that are producing
hot crypto
and to try and make sure that they are willing active contributors. For e.g.,
there was a company that wanted to join, but as far as I could see they were
going to keep their efforts proprietary and just incorporate the Cryptix bits
were there were gaps.
This is a working policy, not dogmatic truth. It's a combination of
logic, experience and paranoia. So, comments eagerly sought.
In particular, I am thinking about
The Cathedral and the Bazaar
which says we should open up.
How can I help someone into the team?
If you know of a person that wishes to "get involved" then try this
strategy:
-
Pick an area that is smallish and not really involved
such as a secret key algorithm or a new packet format.
Start the person on that, and be their interface to the
team. This way, they can concentrate on the code.
-
When the code is done,
submit it into the archives in the normal way.
-
If your friend is still willing to continue, then sponsor the
person into the team, and help with the selection of further
tasks.
Culture
Nope, no culture yet.
Give it another thousand years and some yeast and hops.
A Rant on the License
Follows is another slightly dynamic rant by
iang:
The goal of the Cryptix license is to efficiently
allow freeware crypto to be published.
The license does not change the ownership of the code, that remains vested in
the original owner. So, if your University applies the rule that all student
works are owned by the University (as many do), then this is no problem
as the University can be stated as such: the owner of the code. Although
we might want to better reflect that distinction in the FAQ, which is
where it is currently preferred to list such issues, once, in full.
The Cryptix license also has no effect on the authorship of the code.
The current method is to put the author in the @author tag, as code is
written and changed.
What the license does do is assert copyright on behalf of the owners
(not the authors, I guess), and it names them as the Cryptix Development
Team. This then becomes a device for protecting the code against
attacks. It is also generic and common, which means that we save a lot
of hassle in distribution time: one notice throughout the system means
ease of packaging and ease of understanding by readers.
The motive for using Systemics as the asserter or claimor of the rights
(on behalf of...) has changed over time. Originally Systemics wrote the
code, so it was obvious.
Now, Systemics doesn't write (much of) the code, and doesn't wish to (much).
But there is some value to the group in maintaining a corporate umbrella
over the product. Copyright is normally asserted by corporations and not
individuals for the simple reason of limited liability. That is, if one
of the PGP packets goes astray on the way to NASA's latest launch and the
bright new space station decides its geosync orbit is just north of the
Empire State building, then some questions are going to be asked...
In this case, Systemics makes a useful protective device for the individuals.
Basically, when the squillion-dollar lawsuit lands, Systemics goes under,
all assets sunk, and we all move onto fat talking engagements about how
we sunk the space station. I am somewhat flippant about this because
a. it's unlikely, and b. it's a risk of business, and c. I have dealing
with companies of this nature for years. It's also worth noting that
companies and lawyers know what is going on. The game is familiar to all.
There are other considerations.
Of course this comes at a cost. I gather the DebIan (sp?) crowd of Linux
hackers have recently incorporated (I suppose for these reasons) and it
has cost them a couple of thousand dollars. Which they don't have,
donations kindly accepted :))
There is also a risk to Systemics that the boys at no such agency will
phone up the regulators and put the squeeze on. Fine, Cryptix ends up
squeezed all over the net, and Systemics a. folds, or b. disowns it.
The current benefits to Systemics are of course some useful kudos,
and some possibility of picking up consulting work.
As far as collected revenues go
they have not been worth squabbling over so far :-/
but the future holds more potential.
Indeed, with recent activity, there is
a plausible upside for Systemics, as there
is for the team.
However, it is an open market.
Any member of the team is free to
sell software or consulting or work
based on Cryptix; the license ensures
that everyone has at least an equal
chance of doing well. So Systemics
has to maintain a fair balance, else
risk breaking up the team.
Team-FAQ History
$Log: FAQ.html,v $
Revision 1.13 1999/07/26 15:46:08 iang
elimination of old systemics.com domains, emails, etc.
Revision 1.12 1999/03/05 19:18:40 gelderen
added putty.
Revision 1.11 1999/03/02 20:55:13 gelderen
added SSH comment
Revision 1.10 1998/06/01 19:33:53 iang
- updated the unix ssh cvs access notes
Revision 1.9 1998/04/26 18:58:19 iang
- formatting of revisions (again).
Revision 1.8 1998/04/26 18:56:20 iang
-
Oops, previous was correct, the cryptx repository is odd, not the FAQ.
Revision 1.7 1998/04/26 11:39:21 iang
- dropped cvsroot from home in ssh line for cvs access. wrong.
Revision 1.6 1998/02/17 20:55:05 iang
- correction to Edwin's name
Revision 1.5 1998/02/15 17:56:32 iang
- fixed silly comment
- actually, the new members were added to the project list.
Revision 1.4 1998/02/15 17:53:58 iang
- zox address wrong
Revision 1.3 1998/02/15 17:52:43 iang
- added updates
Revision 1.2 1998/01/31 23:38:31 iang
- moved from ../cryptix/team-FAQ.html
- not at later version, needs to jump to 1.15 or later
Revision 1.11 1998/01/12 21:50:52 hopwood
- Moved history to end of file.
- Fixed some typos, cosmetics.
Revision 1.10 1998/01/11 23:59:00 iang
- added section on how to add new units.
- reformatted history.
Revision 1.9 1998/01/11 17:01:59 iang
- Added substantial section on unit tests.
- Reviewed build.
Revision 1.8 1998/01/09 22:34:34 iang
- added section on 32 character filenames
Revision 1.7 1998/01/06 20:14:24 iang
- team-FAQ: updated to review distribution current practice
- Support: a few updates from post
- Compilation: ?
Revision 1.6 1998/01/02 21:43:52 iang
- updated current projects
Revision 1.5 1998/01/02 21:31:16 iang
- added reference to NIST quest
Revision 1.4 1997/12/30 01:26:19 iang
- added refs for maths algorithms in HB/AC Menezes to wish list
Revision 1.3 1997/12/23 19:20:17 iang
- press release finished off, as published, mostly
Revision 1.2 1997/12/23 15:31:41 iang
- Copied in my working team FAQ to this location, which is at least visible.
- Many Many Changes, this one was well out of date.
Copyright © 1997 Systemics Ltd
on behalf of the Cryptix Development Team.
All rights reserved.
Cryptix is a trademark of Systemics Ltd.