Pervasive DataTools Weblog

DataTools HowTo: Transpose Data from Vertical to Horizontal Orientation

jterrell — Fri, 05 Mar 2010 17:22:56 -0600

Question:

I have a source text file that contains two fields/columns. In the first column there are product SKUs, each of which is repeated on multiple rows. The second column contains the SIZEs of each of the products that I have in stock. For example, if I have 6 different SIZEs of one SKU, there are 6 rows for that SKU in my source file. How can I combine the data into a target text file that contains just one row for each SKU and each of the available SIZEs on that same row and in size order and separated by a vertical bar?

The format of the source file looks like this:

"SKU","SIZE"

"1111A",8

"1111A",6

"1111A",16

"1111A",12

"1112B",13

"1112B",8

"1112B",5

"1112B",16

"1112B",12

"1113C",6

"1113C",8

"1113C",14

I would like the format of the target file to look like this:

1111A 6|8|12|16

1112B 5|8|12|13|16

1113C 6|8|14

Answer:

Using the Data Loader to CSV Text or the Data Loader to Fixed Text, you can transform the format of the data exactly as you have described and illustrated.

Objective

There is one objective in this sample transformation map, as follows:

DataTools HowTo: Solve an "Overflow (6)" Error

jterrell — Wed, 03 Mar 2010 15:59:47 -0600

Question:

For the past 2 days I get an "Overflow (6)" error message when I launch my DataTools product for the first time each morning. What is causing the error, and how can it be fixed?

Answer:

The Cause

The error is usually the result of “corrupted” window dimension/location values in the “settings” file for the product. This is sometimes caused when the application is closed (intentionally or not) with one or more of the application's windows in a position that is outside the visual boundaries of your computer screen.

The Solution

The solution to the problem depends on which of the DataTools products is affected. Each product within a "category" (i.e., Extractors, Loaders, Parsers, etc.) creates/edits and uses a unique "settings" file each time the product is launched and closed. The "settings" files reside in the following path, depending on the operating system of your workstation on which DataTools is installed, as follows:

On Windows XP:

C:\Documents and Settings\[YOUR_USERNAME]\Application Data\Pervasive\DataTools9\Common\Settings

The following list includes some of the DataTools product category names and the associated "settings" file name:

Data Joiner - DataJoiner.xml
Data Loaders - DataLoader_Settings.xml
Data Extractors - DataExtractor_Settings.xml
Data Parsers - DataParseSettings.xml
Data Parser for Unstructured Text - ExtractorSchema.xml
Data Viewer - Browse.xml

IMPORTANT NOTE: For the Data Parser for Unstructured Text back up the extractor900.mdb file prior to deleting the settings file. Otherwise, you will lose any/all saved extraction scripts. The extractor900.mdb file resides in this folder path (Windows XP): C:\Documents and Settings\jterrell\Application Data\Pervasive\DataTools9\Common

DataTools HowTo: Manually Parse a C-ISAM Data File

jterrell — Thu, 18 Jun 2009 16:28:45 -0500

Question:

I have a C-ISAM file that contains packed and binary data. How can I parse and define the structure of the file and export the unpacked data to a CSV text file format?

Answer:

Using the Data Parser for C-ISAM, you can define the structure of the data records by using the visual parser window, and once the data is parsed and unpacked, export the data to a CSV text file.

Objectives

There are two objectives in this DataTools HowTo, as follows:

Use the Data Parser for C-ISAM to define the schema/structure of a fixed-length C-ISAM data file
Export the unpacked data to a CSV text file format

Skill Level

Basic to Intermediate

Skill Set

Data Parser
Built-in visual parsing interface
Basic understanding of C-ISAM data storage formats

Design Considerations

The principal design consideration is to follow the procedural steps below in the order in which they are specified in this HowTo. A secondary consideration is to follow the same basic procedural steps, and in the same order, when you begin to define the structure/schema of your own data file.

The DataTools product used in this DataTools HowTo is the Data Parser for C-ISAM.

The sample C-ISAM data file used in this HowTo is a simple fixed-length data file that contains character, binary, and packed data. The C-ISAM connector requires that the data file and its matching index file reside in the folder on your workstation. The sample data and index files (SrcCISAM.dat and SrcCISAM.idx) were NOT copied to your workstation when DataTools was installed, but you can obtain the sample files by posting a request in the user forum for the Data Parsers. We will zip and email the files to you.

DataTools HowTo: Manually Parse a Btrieve Data File

jterrell — Tue, 16 Jun 2009 13:39:55 -0500

Question:

I have a Btrieve data file that contains packed and binary data. How can I parse and define the structure of the file and export the unpacked data to a CSV text file format?

Answer:

Using the Data Parser for Btrieve, you can define the structure of the data records by using the visual parser window, and once the data is parsed and unpacked, export the data to a CSV text file.

Objective

There are two objectives in this DataTools HowTo, as follows:

Use the Data Parser for Btrieve to define the schema/structure of a fixed-length Btrieve data file
Export the unpacked data to a CSV text file format

Skill Level

Basic to Intermediate

Skill Set

Data Parser
Built-in visual parsing interface
Basic understanding of Btrieve and packed data storage formats

Design Considerations

NOTE: Before you begin the procedures in this HowTo, you must have a Btrieve or Pervasive.SQL Engine installed and configured on your system, and DataTools must be able to detect it. Refer to this page of our web site for details:

DataTools HowTo: Resolve an Error in the Btrieve Parser - Client Missing

The principal design consideration is to follow the procedural steps below in the order in which they are specified in this DataTools HowTo.

The DataTools product used in this HowTo is the Data Parser for Btrieve.

DataTools HowTo: Manually Parse a Binary Data File

jterrell — Mon, 15 Jun 2009 16:45:13 -0500

Question:

I have a source file that contains packed and binary data. How can I parse and define the structure of the file and export the unpacked data to a CSV text file format?

Answer:

Using the Data Parser for Binary, you can define the structure of the data records by using the visual parser window, and once the data is parsed and unpacked, export the data to a CSV text file.

Objective

There are two objectives in this DataTools HowTo, as follows:

Use the Data Parser for Binary to define the schema/structure of a fixed-length data file
Export the unpacked data to a CSV text file format

Skill Level

Basic to Intermediate

Skill Set

Data Parser
Built-in visual parsing interface
Basic understanding of binary and packed data storage formats

Design Considerations

The DataTools product used in this DataTools HowTo is the Data Parser for Binary.

The sample binary data file used in this HowTo is a simple fixed-length data file that contains binary and packed data. The sample data file name is TUTOR4.BIN and it was copied to this folder when DataTools was installed on your workstation, assuming the product was installed in the default location:

C:\Program Files\Pervasive\DataTools9\Common

Procedure

This tutorial is divided into the following tasks, which you should complete in the order shown:

DataTools HowTo: Standardize Various Date Formats

jterrell — Thu, 11 Jun 2009 12:42:22 -0500

Question:

I have a source text file that contains a field in which there are dates stored in five different character string date formats. How can I standardize the dates to a single format?

Answer:

Using Map Designer in any of the Pervasive DataTools Data Extractors or Data Loaders, you can transform a variety of date formats into one standardized date format in your target file or table.

Objective

There is one objective in this sample transformation map, as follows:

Use the Rapid Flow Integration Language (RIFL) to write an expression in a target data field to transform a variety of date formats from the source file into a one standardized date format in the target data file.

Skill Level

Basic

Skill Set

Map Designer
Basic RIFL Scripting
Basic understanding of date formats and the importance of storing dates in a single, standard format

Design Considerations

The principal design consideration is to choose the date format that fits your business process requirements. Following this decision, the task is to write a RIFL expression that converts any date in the source file to the chosen standardized target date format.

The DataTools product used in this sample transformation map is the Data Loader to CSV Text.

The selected standardized date format in this sample is mm/dd/yyyy.

Procedure

Launch any of the DataTools Data Loader to CSV Text. Create a new transformation map, connect to the source and target data files, and perform the data field mapping and expressions, as follows:

Source Connection

The Source Connection tab is where you specify the source connector, select the source data file or table, and specify any special properties about your source data file. For this sample select the following:

DataTools HowTo: Write Rejected Records to a Reject File

jterrell — Thu, 14 May 2009 14:46:30 -0500

Question:

There are records in my source file that I do not want to convert to my target file because they do not meet specific requirements. I know I can filter these records out completely, but can they be written to a different target file in the same map?

Answer:

Using Map Designer in any of the Pervasive DataTools Data Extractors or Data Loaders, you can filter records that match or fail specified criteria and write these records to a reject file. The resulting records in the reject file can be inspected, thus allowing you to determine how to handle those records for further processing, or not.

Objectives

There are two objectives in this sample transformation map, as follows:

Use the Rapid Flow Integration Language (RIFL) to write an expression in the Account No target field to prevent certain data records from being written to the target data file.
Use the reject records subsystem to write the rejected records to a reject file where the rejected records can be reviewed manually and/or create a separate transformation map to correct the situation(s) that caused the records to be rejected.

Skill Level

Basic

Skill Set

Map Designer
Basic RIFL Scripting
Event Handlers and Actions

Design Considerations

This transformation map selects records based on values in a data field in the source file. The field of interest is Account No. The goal is to write to the target only the source records that fall within a specified range of account numbers. We accomplish this goal by writing a simple RIFL expression in the Account No target field. The records from the source file that do not fall within the specified range of account numbers will be written to a "reject file".

DataTools HowTo: Use a COBOL Copybook to Define Fields

jterrell — Wed, 13 May 2009 13:17:51 -0500

Question:

Can I use a COBOL copybook (FD) to define the fields in my COBOL data file? And if so, how?

Answer:

Yes, each of the following DataTools Data Parsers includes a feature that allows you to use a COBOL copybook to parse COBOL data. Detailed steps, including screenshots, are provided for you in the attached PDF document.

Adobe Reader is required to read the attached document.

DataTools HowTo: Transpose Data From Horizontal to Vertical Orientation

jterrell — Fri, 01 May 2009 12:21:28 -0500

Question:

Is there a way to change the orientation of my data from rows to columns?

Answer:

This "DataTools HowTo" contains an example of a data transformation showing how data stored in a row format in your source data file can be transposed into a columnar format in your target data file.

The DataTools product used in this sample transformation map is the Data Loader to Fixed Text.

Source Connection

The source connector is ASCII Delimited, and the source file is TUTOR1.ASC. This is a sample CSV text file that resides in the main DataTools installation folder on your workstation.

Target Connection

The target connector is ASCII Fixed, and the target file that will be created after running the transformation map will be TRG_TRANSPOSE.ASC. You may write the target file to any folder to which you have access and permissions.

Transformation and Map Properties

To accomplish the horizontal-to-vertical transformation, the Event Handlers and Event Actions must be used in the Transformation and Map Properties window. To access this window click on the View menu and select Transformation and Map Properties from the menu options.

Four Event Actions are created in the Transformation Map Properties. The following describes how to create these Events.

Pervasive Data Inspectors Released!

jterrell — Fri, 10 Apr 2009 15:16:43 -0500

The Pervasive DataTools Team is proud to announce the release of a new line of DataTools products - Data Inspectors.

There are 22 Data Inspector products, each of which connects to a specific application, database, or data file. Each Data Inspector runs an analysis on user-selected data fields and data records in your data source and returns a tabular (row and column) and a graphical (pie chart) analysis of the Distinct Values for each data field on which you choose to run the data analysis.

The Data Inspector is a valuable data tool that can assist you with data analysis, data cleansing, and data transformation requirements.

Documentation for the Data Inspectors is available on the Online Help page in the Support area of our web site.

Check out the new Data Inspectors and give us your feedback via the Product Feedback page of our site. We look forward to hearing from you!

- Pervasive DataTools Team

DataTools HowTo: Resolve Btrieve Error 161

jterrell — Mon, 09 Mar 2009 11:12:00 -0500

Question:

When I try to connect to a Btrieve data file, I get an error 161.

Answer:

A Btrieve Error 161 indicates that the Pervasive.SQL user Count is exceeded. You may need to check with the DBA and request that any "duplicate" login(s) be shutdown, or purchase sufficient licenses for the number of users who need to simultaneously access the Pervasive.SQL database.

DataTools v9 is Released!

jterrell — Thu, 05 Mar 2009 19:03:49 -0600

The Pervasive DataTools Team is proud to announce the release of DataTools version 9!

Some of the product names have changed, as follows:

* Data UpLoaders v8 ==> Data Loaders v9
* Data DownLoaders v8 ==> Data Extractors v9
* Data Extractor for Unstructured Text v8 ==> Data Parser for Unstructured Text v9

The functionality of the above renamed products will not change from v8 to v9.

The new version 9 products will include the following enhancements:

* Support for new versions of applications, databases, and data file formats
* Improved support for Internet connections via a proxy server
* Delivery of new and updated metadata via our web service
* ...and more

The user forums will be enhanced to include one "set" of forums for the v8 products and one "set" of forums for the v9 products. A new forum entitled "DataTools v9 Upgrade" will be added to address your version transition questions and issues.

How Will this Upgrade Affect You?

One of the most important changes that comes with this upgrade is that you can no longer install and run both Pervasive DataTools v9 AND Pervasive Data Integrator v9 on the same workstation. This limitation is due to the fact that both DataTools v9 and Data Integrator v9 write the same keys to the Windows Registry. If you wish to run both products, you must install each on a different workstation.

First Time Users

If you are a first-time user of DataTools, and have never installed DataTools v8, you will not be affected by this upgrade except immediate access to the new features. However, please see the previous paragraph regarding the installation of DataTools and Data Integrator on the same workstation.

Returning Users

Pervasive DataTools v8 product licenses will no longer be available. If you are currently using a license for one or more of the DataTools v8 products, please see the following important information: