Computrac and MetaStock File Formats

This is a description of the file format called Computrac or MetaStock format. It is intended as background information for software design. The Computrac system appeared early in the IBM PC days, in about 1984 and was written in Microsoft Basic. The company was later sold to Reuters and has lost its separate identity. The Computrac software was licensed to Stratagem Software International (phone 504-885-7353). It currently is named SmartTrader Professional V5.2000. It recently was updated for Year 2000 compatibility.

Many technical analysis systems read Computrac format files, either as their primary format or as an alternative. The only significant compatibility problem is the need to convert the early Microsoft Basic floating point formatted number to the current IEEE floating point format, standard for personal computers.

Each directory holds a group of related security files. The files are named by a number representing the order in which they were created: F1, F2, F3 and so on. Deletions are allowed, so some F# may be skipped. This method overcomes the limit by MS-DOS on file name length and characters. An added file, called MASTER, is located in each directory to associate each file with its matching security name and NASDAQ trading symbol. An application which wished to read a specific data file must lookup the F# by referring to the MASTER file.

The Computrac method uses two data files per security. One holds a descriptor of the data fields and the other is a time series of prices, volume, open interest and similar data.

Directories may be nested. Thus, the directory STOCKS may contain a sequence of F1, F2 and F3 data files for three securities. In addition it could contain a MS-DOS sub-directory FOREIGN. Within FOREIGN could appear F1, F2 and F3 holding data for three foreign securities.

In the original Computrac system the file MASTER contains an entry for each F# file and an entry for each sub-directory appearing within that directory. Thus a tree structure of nested directories may be formed. In MetaStock and later system this feature of sub-directories is ignored. If the tree structure is desired directory maintenance must be done within Computrac.

A second difference between the original Computrac format and current use is that Computrac allows the user to define the name and scale factors for each field in the data files. The default values are Date, High, Low, Close and Vol. For example, if market statistics are held the user may create such a file and rename the fields to Date, Adv, Dec, Up, Dn, Vol. MetaStock and other common systems do not support this feature. They ignore the F#.dop format file and assume the fields are price, volume and open interest depending on the number of fields present. This is further discussed below in the section Scale Factors.

The Computrac/MetaStock format has three structural levels. The top level is a MASTER file which lists the names, trading symbols and miscellaneous information about securities data files of the directory in which it is located. EMASTER is a similar file (Extended Master file) maintained by MetaStock as an extension to the original Computrac format. There is a limit of 254 security files per directory in the original Computrac form.

The second level is the "Fx.dop" file which gives the data fields and scale factors for each data file. Again, this feature of the original Computrac system is not supported by current software vendors.

The third level is the "Fx.dat" file which contains a time series of prices for each security. Under the Computrac system the data fields may be customize for each security. This feature is not supported by later systems (MetaStock).

The 'x' in each file name is an integer from 1 to 254. Thus, the securities in a directory appear as: F1.dop & F1.dat and so on. (i.e. F1.dop, F1.dat, ... F254.dop, F254.dat)

The MASTER file in each directory has one record per security. Each record gives the file number "Fx" for the security along with its text name, trading symbol, time base and update information. This file is organized a 53 bytes in a fixed length field format. The fields may be ASCII characters, binary integers or floating point number in an old Microsoft Basic Floating point format (MBF). It is necessary to convert these floating point number into the current IEEE floating point format used in contemporary computers.

The MASTER file layout, record 2 onward:

Field Name Format Start Size Function

File Number UB 1 1 The n value of file names Fn

Type UW 2 2 Computrac file type = $e0

Length UB 4 1 Record length

Fields UB 5 1 Fields per record in Fn.dat file

Reserved1 6 2 Contains $00 $00

Security A 8 16 Security name in ASCII blank padded

Reserved2 24 1 Contains $00

Vflag A 25 1 $00 Version 2.8 flag

First Date MBF 26 4 First date in Fn.dat file

Last Date MBF 30 4 Last date in Fn.dat file

Period A 34 1 Time period for records: IDWMQY

Time UW 35 2 Intraday time Base, $00 $00

Symbol A 37 14 Trading symbol, blank padded

Reserved3 51 1 Contains $20

AutoRun A 52 1 ASCII '*' for autorun

Reserved4 53 1 Contains $00

The formats are:

UB = unsigned byte 'unit8'

UW = unsigned word 'uint16'

A = ASCII characters

MBF = Microsoft Basic Floating point in 4 bytes

The first record in the MASTER file specifies the structure of the remaining MASTER file, records 2 onward.

This first record is 53 bytes long organized as:

Field Name Format Start Size Function

Number of Files UW 1 2 Number of files in MASTER

Next file UW 3 2 Number to assign to next new Fn file

Reserved5 UB 4 45 Contains $00

Unknown MBF 49 4 Unknown value

The general access method would be to:

1. Open and read the MASTER file.
2. Determine the number of records in MASTER
3. Skip to the start of MASTER record two.
4. For each 53 byte record in MASTER, determine File Number of each security file, its name, trading symbol and the number of fields it contains.
5. Construct each security file name as "Fx.dat."
6. Using the security file name, open the security files by a call to the operating system as desired. From specific F#.dat files read the proper span of data based on the number of fields, convert the data to IEEE format and display according to the format in the Fn.dop file (if used).

The EMASTER file was added by MetaStock. It has a simpler structure that MASTER and used IEEE short floating point numbers

The first record in the EMASTER file specifies the structure of the remaining MASTER file, records 2 onward.

This first record is 192 bytes long organized as:

Field Name Format Start Size Function

Number of Files UW 1 2 Number of files in EMASTER

Last file UW 3 2 Last assigned Fn file

Reserved5 UB 4 188 Contains $00

The EMASTER file layout, record 2 onward:

Field Name Format Start Size Function

ID code A 1 2 ASCII "30", $34 $31

File Number UB 3 1 File number for Fn.dat

Filler1 A 4 3

Fields UB 7 1 Number of 4 byte data fields

Filler2 A 8 2

AutoRun A 10 1 Either $00 or "*" for autorun

Filler3 A 11 1

Symbol A 12 14 Stock symbol, null padded

Filler4 A 26 7

Name A 33 16 Security name, null padded

Fill5 A 49 12

Time Frame A 61 1 Ascii: DWM

Fill6 A 62 3

First Date CVS 65 4 First date of data in Fn.dat 'yymmdd'

Fill7 A 69 4

Last Date CVS 73 4 Last date of data in Fn.dat, 'yymmdd'

Fill8 A 77 50 unknown

First Dt Long CVL 127 4 First Date, long format YYYYMMDD

Fill9 131 1 unknown

Dividend Date CVL 132 4 Date of last dividend CVL format

Dividend Rate CVS 136 4 Dividend adjustment value CVL

Fill10 140 53 unknown

Notes:

CVS format is 4 byte single precision real

CVL format is 4 byte long integer

The ".dop" file is an ASCII file with variable length text records. It is purpose is to allow the user to customize the data fields for any security and specify input/output precision for each field. This is an outstanding feature of Computrac but is not observed or maintained by other vendors.

Each ".dop" record specifies one field of the ".dat" file. Each record ends in <cr><lf>. For example this "F1.dop" file:

"DATE",0,0<CR><LF>

"HIGH",2,2<CR><LF>

"LOW",2,2<CR><LF>

"CLOSE",2,2<CR><LF>

"VOL",0,0<CR><LF>

Specifies the usual 5 field data format. Note that the ASCII entries are delimited by double quotes, there is no comma after the second zero and each record ends with ASCII $0D and $0A (<CR><LF>). The first ASCII number is the number of decimal places or fraction format displayed on screen. The second ASCII number is the number of decimal places or fraction expected upon user editing input. Example: "HIGH",2,3<CR><LF> would specify two decimals (XX.XX) when the data is displayed and three (XX.XXX) when the data is input.

The two numeric values in the ".dop" file specify the decimal location or fraction size for display and input. The factors are:

Factor Display Scale Example

1000 1000s omit 3 zeros 100,000 = 100

100 100 omit 2 zeros 100,000 = 1000

10 10 omit 1 zero 100,000 = 10000

4 .0001 show 4 decimals 45.1234

3 .001 show 3 decimals 45.123

2 .01 show 2 decimals 45.12

1 .1 show 1 decimal 45.1

0 integer 45

-1 1/2 digit is in 1/2s 12 ½ = 12^1

-2 1/4 12 ¾ = 13^3

-3 1/8 12-7/8 = 12^7

-4 1/16 12-3/16 = 12^3

-5 1/32 12-5/32 = 12^5

-6 1/64

-7 1/128

The Computrac data editor depends on the .dop file for decimal point location. These values are used to set decimal locations and numeric type for input editing and display. They have no effect on data being downloaded or converted from external sources. Thus, the user inputs only numeric values with no puncutuation ("." and ","). If the security file being edited specifies 2 decimals for input then inputing 12345 will product 123.45 in the sorted data. The .dop files are apparently ignored by MetaStock. Omega SuperCharts mentions them as being used to specify data order if not in the expected order (Date, Open, High, Low, Close, Volume).

Note that MetaStock does not utilize this information nor allow custom data fields. It assumes all input and display is stock price data in one of these formats:

5 fields: Date, High, Low, Close, Volume

6 fields: Date, Open , High, Low, Close, Volume

7 fields: Date, Open, High, Low, Close, Volume, Open Interest.

Until MetaStock V6.5 the MetaStock Downloader created and maintained the ".dop" file. It appears that V6.52 and later no longer creates or maintains this ".dop" file. Thus exact compatibility with Computrac and SmartTrader (the successor to Computrac) has been lost. To maintain that compatibility, new securities files should be created with Computrac.

The file holding the security price series has a name "Fn.dat" with "Fn" incommon between the .dat and .dop files. The actual security name, symbol, date range and number of fields per record appears in the MASTER file. Each data file record contains (at most) fields for the date, and the price open, high, low, close, volume, and open interest. Computrac supports from 4 to 7 fields per record. Date is required. The most common arrangement is five fields: date, high, low, close, volume. The next most common, especially for commodities, is seven field: date, open, high, low, close, volume, open interest.

Computrac also supports custom naming and field formats for number of decimal places. MetaStock doesn't support this ability and also forces the volume and open interest fields to be integer value, although stored as floating point numbers.

Dates are expressed as floating point integers. Date from 1900 to 1999 have the format YYMMDD using two digits for Year, Month and Day. Thus Jan. 23 of 1953 would appear as a floating point integer 530123.

Dates after Jan. 1, 2000 have a leading "Century" digit in the form CYYMMDD. Following the convention only the last two digits of the year appear in position YY. Thus, December 17 of 2006 appears as 1061217

Most current computer systems express the date as a day number from some starting date, say Jan.1, 1964. To convert one usually takes the Computrac number and breaks it into year, month and day values. These are passed to the host to convert into the day number.

Day number = Modulo(CYYMMDD,100) (remainer after dividing by 100)

Month = Modulo((CYYMMDD/100),100) (remainder after division of CYYMM by 100)

Year = Modulo( CYYMMDD/10000),100) (remainder after extracting CYY)

Century = Modulo(CYYMMDD,1000000) (remainder after extracting C)

This is passed to the host program in the form:

Host value for (Year, Month, Day) = ((Century*100)+1900+YY,MM,DD)

A key part is the conversion to the Computrac/Metastock format from the old Microsoft Basic floating point format. See also "The Revolutionary Guide To Q Basic, by Dyakonov, Yemelchenkov, Munerman & Samolytova, Wrox Press, 1996.

In the Computrac system floating point numbers are represented in the old Microsoft Basic Floating point format. It consists of 4 eight bits bytes, called single precision. The layout of an MBF number is:

Bit 31 24 23|22 16 15 8 7 0

X EEEEEEE EMMMMMMM MMMMMMMM MMMMMMMM

^H to left of bit 22

Components:

X = sign bit

E = 8 bit exponent

M = 23 bit mantissa

H = "hidden bit" implicitly = 1

By definition, the value of zero is all bits zero in both MBF and IEEE. This convention allows the numeric value of zero to represent a logic "false" and any non-zero value to represent "true."

The exponent is a twos complement, 9 bit, signed binary number. The mantissa is a 23 bit signed binary number. When being read from or stored to memory the mantissa is left normalized. This means that the mantissa is left shifted until a 1 bit appears in the left most bit position and the exponent is scaled to match. Since this leftmost bit is always 1 it is not stored. This is the so-called "hidden bit" noted above as "H." This bit is restored during the conversion process. This method allows a larger numeric range within 4 bytes or 32 bits.

Microsoft's Qbasic has several conversion routines:

CVSMBF 4 byte string to single precision

CVDMBF 8 byte string to double precision

MKSMBF$ single precision to 4 byte string

MKDMBF$ double precision to 8 byte string

The IEEE Floating Point (noted as IEEE) was developed in the mid-1980s. It was originally supported in software, then by a separate chip (8087) and currently by the computer central processor (80486, Pentium).

The IEEE (double precision or extended format) format consists of 8 bytes laid out as:

Bit 64|62 52 | 51 0

X EEEEEEEEEEEEEEE SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

X= sign of significand, 0=positive, 1=negative

E = 10 bit biased exponent

S = 52 bit significand as sign magnitude number, observing sign X

The byte boundaries have been omitted. The exponent is a biased value which is not intuitively obvious. It has a range of $FF, the largest positive exponent, decimal 2^127, down to $81 representing an exponent of 2^1, to $80 representing 2^0, continuing to $01 the smallest negative exponent (2^-127). The final exponent of $00 is reserved as part of the number consisting of all zero bits which is defined as the number zero. The format also allows for NAN (not a number) and infinity (?).

The significand (called mantissa in earlier floating point methods) is not a two's complement number. It is in sign magnitude format. The major difference between it and twos complement is that sign magnitude has two values for zero (no bits set and all bits set) and twos complement represents zero only as zero.

The IEEE format likewise has a hidden bit at the left of bit 51. The number is always scaled to produce a hidden bit of one (1) and thus it need not be shown and is therefore "hidden."

The complexities of the IEEE format are mostly handled in hardware which simplifies our conversion routines.

This conversion process accepts four bytes of MBF 'input' producing eight bytes as IEEE floating point 'output.' For discussion, the input and output bits are numbered from the least significant bit zero on the right, upward, to either bit 31 (MBF) or 63 (IEEE) on the left. Note that the MatLab code below numbers the input as bits from 32 down to 1.

1. If the input is zero bits, then return zero bits as the output and terminate. Otherwise...
2. AND the input with decimal 16777215, which is 2^24-1. This leaves the right most 24 bits with higher order bits set to zeros.
3. SET BIT 23 to 1. This restores the implied hidden bit by writing over bit 23 in the current value (not the input). The original bit 23, the mantissa sign, will be processed later.
4. Maintain the working value as TEMP-1. In IEEE terms this is the significand.
5. AND the original input by decimal 4278190080, which is XOR(2^32-1,2^24-1). This selects the high order 8 bits with lower 24 bits set to zeros, which selects the exponent sign bit followed by the 7 bit exponent value.
6. SHIFT 24 bit positions to the right. This leaves the input exponent as a bitwise binary integer in the low bit positions, 7-0.
7. SUBTRACT the exponent bias value of 152. This conversion is derived from the signing of the MBF exponent and the exponent offsetting method used within the IEEE format. It value is not obvious but does properly capture the translation of an MBF exponent to IEEE exponent.
8. Raise 2 to the power of the integer value from step 7. This is the IEEE exponent.
9. Multiply this value by TEMP-1 (from Step 4). The IEEE significand is now scaled by the IEEE exponent.
10. From the original input value TEST bit 23 (sign bit of mantissa). If it is set (value=1) then multiply the result from Step 9 by -1. This adjusts the sign of the resulting IEEE number.
11. Return this result as 'output.'

function output=MBF2IEEE(input);

% Convert 32 bit Microsoft Basic Float into IEEE format for MatLab

% If input is an array, all values will be converted.

% Note that MatLab numbers bits from 24 (high end on left) down

% to bit 1 (low end on right). This differs from the narrative

% above which numbers them from 23 down to 0.

% mask1=16777215 ; % 2^24-1 ; % bottom 24 bits holds the mantissa

% mask2=4278190080 ; % bitxor(2^32-1,mask1) ; % top 8 bits holds the exponent

% sign=bitget(input,24) ; % hi bit in mantissa

% mantissa=bitset(bitand(input,mask1),24) ; % restore hidden bit

% exponent=bitshift(bitand(input,mask2),-24)-152 ; % scale exponent

% sign=((bitget(input,24)==0)*2)-1 ; % +1 for zero or positive, -1 for negative

% zeros=(input~=0) ; % 0 for zero values else +1

% output=mantissa*2^exponent*zeros*sign, as done below:

output= bitset(bitand(input,16777215),24)...

.*(power(2,(bitshift(bitand(input,4278190080),-24)-152)))...

.*(input~=0).*(((bitget(input,24)==0)*2)-1) ;

return;

The following C routines specify the declarations for the Computrac data formats and have default values.

One major nuisance is that these files all use an early Microsoft Basic floating point number format. Conversion routines appear below.

================================

typedef unsigned char u_char;

typedef unsigned short u_short;

/*

* MASTER file description which describes the directory contents

* floats are in Microsoft Basic format

* strings are fixed length items padded with spaces, not null terminated */

struct rec_1 {

u_short num_files; /* number of files MASTER contains */

u_short file_num; /* next file number to use (highest F# used) */

char zeroes[49];

};

struct rec_2to255 { /* description of data files */

u_char file_num; /* file #, i.e., F# */

char file_type[2]; /* CT file type = 0'e' (5 or 7 flds) */

u_char rec_len; /* record length in bytes (4 x num_fields)*/

u_char num_fields; /* number of 4-byte fields in each record*/

char reserved1[2]; /* in the data file */

char issue_name[16]; /* stock name */

char reserved2;

char CT_v2_8_flag; /* if CT ver. 2.8, 'Y'; otherwise, anything else */

float first_date; /* yymmdd */

float last_date;

char time_frame; /* data format: 'I'(IDA)/'W'/'Q'/'D'/'M'/'Y' */

u_short ida_time; /* intraday (IDA) time base */

char symbol[14]; /* stock symbol */

char reserved3; /* MetaStock reserved2: must be a space */

char flag; /* ' ' or '*' for autorun */

char reserved4;

};

/*

* EMASTER data structure, the Extended MASTER file specific to MetaStock

* floats are in IEEE format

* strings are padded with nulls

*/

struct emashdr {

u_short num_files; /* number of files in EMASTER */

u_short file_num; /* last (highest) file number */

char stuff[188];

};

struct emasdat {

char asc30[2]; /* "30" */

u_char file_num; /* file number F# */

char fill1[3];

u_char num_fields; /* number of 4-byte data fields */

char fill2[2];

char flag; /* ' ' or '*' for autorun */

char fill3;

char symbol[14]; /* stock symbol */

char fill4[7];

char issue_name[16]; /* stock name */

char fill5[12];

char time_frame; /* data format: 'D'/'W'/'M'/ etc. */

char fill6[3];

float first_date; /* yymmdd */

char fill7[4];

float last_date;

char fill8[116];

};

/* This is the DATA file description for "fn. dat" files with seven data fields */

struct dathdr7 {

u_short max_recs; /* 0 ==> unlimited size */

u_short last_rec; /* dathdr7 = 1; ctdata7 starts with 2 */

char zeroes[24];

};

struct ctdata7 {

float date;

float open;

float high;

float low;

float close;

float volume;

float op_int;

};

/* This is the DATA file description for "fn.dat" with five data fields */

struct dathdr5 {

u_short max_recs;

u_short last_rec;

char zeroes[16];

};

struct ctdata5 {

float date;

float high;

float low;

float close;

float volume;

};

/* IEEE floating point format to Microsoft Basic floating point format */

typedef unsigned short u_short;

typedef unsigned long u_long;

int fieee2msbin(float *src, float *dst) {

union {

float a;

u_long b;

} c;

u_short mantissa;

u_short exponent;

c.a = *src;

if (c.b) { /* not zero */

mantissa = c.b >> 16;

exponent = ((mantissa << 1) & 0xff00) + 0x0200;

if (exponent & 0x8000 != (mantissa << 1) & 0x8000)

return 1; /* exponent overflow */

mantissa = mantissa & 0x7f | (mantissa >> 8) & 0x80; /* move sign */

mantissa |= exponent;

c.b = c.b & 0xffff | (long)mantissa << 16;

}

*dst = c.a;

return 0;

}

/* Microsoft Basic floating point format to IEEE floating point format */

int fmsbin2ieee(float *src, float *dst) {

union {

float a;

u_long b;

} c;

u_short mantissa;

u_short exponent;

c.a = *src;

if (c.b) { /* not zero */

mantissa = c.b >> 16;

exp = (mantissa & 0xff00) - 0x0200;

if (exponent & 0x8000 != mantissa & 0x8000)

return 1; /* exponent overflow */

mantissa = mantissa & 0x7f | (mantissa << 8) & 0x8000; /* move sign */

mantissa |= exponent >> 1;

c.b = c.b & 0xffff | (long)mantissa << 16;

}

*dst = c.a;

return 0;

}