[TheOutcaste]

Ok, test file is 52,620,043 bytes, 700,000 lines with one header line.
Elapsed times are displayed below the code I ran:

Using a hard coded Header line:

@Echo off
echo.%time%>add1.log
::Set Output file name here
Set _f{1}=Combined.txt
If EXIST "%_f{1}%" Del "%_f{1}%"
::Output Header to temp file
>%temp%\_f{0} Echo."Name":"Street Address":"City":"St":"Zip":"Filename"
::Read lines from each file excluding the batch file and excluding the header line
::Output to temp file adding :"filename" to end of line
For /F "tokens=*" %%A In ('dir /b /a-d /o:n ^|Find /I /V "%~n0"') Do (
For /F "usebackq skip=1 delims=" %%B In ("%%A") Do Echo.%%B:"%%A">>%temp%\_f{0}
)
Move %temp%\_f{0} "%_f{1}%"
For /L %%A In (0,1,1) Do Set _t%%A=
echo.%time%>>add1.log

0:37:26:53

Reading one file to extract the header line:

@Echo off
echo.%time%>add1.log
::Set Output file name here
Set _f{1}=Combined.txt
If EXIST "%_f{1}%" Del "%_f{1}%"
::Gets first filename in alphabetical order, excluding the batch file
For /F "tokens=*" %%A In ('dir /b /a-d /o:-n ^|Find /I /V "%~n0"') Do Set _t0=%%A
::Read Header from first line in first file
For /F "usebackq tokens=*" %%A In (`Find /V /N "" "%_t0%" ^|Findstr /B /C:"[1]"`) Do Set _t1=%%A
::Output Header to temp file
>%temp%\_f{0} Echo.%_t1:~3%:"Filename"
::Read lines from each file excluding the batch file and excluding the header line
::Output to temp file adding :"filename" to end of line
For /F "tokens=*" %%A In ('dir /b /a-d /o:n ^|Find /I /V "%~n0"') Do (
For /F "usebackq skip=1 delims=" %%B In ("%%A") Do Echo.%%B:"%%A">>%temp%\_f{0}
)
Move %temp%\_f{0} "%_f{1}%"
For /L %%A In (0,1,1) Do Set _t%%A=
echo.%time%>>add1.log

0:38:41:90

And this is Devil's code.

@echo off
echo.%time%>add1.log
setlocal enabledelayedexpansion
set tp=c:\tp.txt
if exist tmpfile.txt del /q tmpfile.txt
(for /f "tokens=*" %%a in ('dir /b /a-d *.txt') do (
   (for /f "tokens=*" %%i in ('cmd /c for /f "usebackq tokens=*" %%j in ("%%a"^) do echo %%j^^^& exit') do set f=%%i)
     (for /f "skip=1 usebackq delims=" %%b in ("%%~dpnxa") do (
     echo %%b:%%~nxa>> "%tp%"
  ))
))

echo %f%:"Filename">tmpfile.txt
type "%tp%" >>tmpfile.txt
del /q "%tp%"
echo.%time%>>add1.log

42:28:04

I was very surprised that Devil's code took longer. I was running WMP10 listening to music and browsing the forums while I ran the tests on my two files. And during the first test, a scheduled backup was running (batch runs on another PC to back up this one). While Devil's code was running, I was not running WMP, was just browsing.

Hard coding the header line vs reading the file didn't make much difference, an extra 80 seconds or so. The slow part seems to be tacking on the filename and writing it to disk.

I'll try this again with two files (2nd will be copy of first), see if the times roughly double, and repeat this test, just to see if the times are consistant.

Jerry

[TheOutcaste]

And just for grins, here's the data file I've been using. Just take the 10 address lines and copy and paste until there are 700,000 lines of addresses:

"Name":"Street Address":"City":"St":"Zip"
"TechGuy Inc":"[PO] Box 268":"Waynesboro":"PA":"17268"
"John Smith":"128 SE Main St":"Seatle":"WA":"98687"
"RALPH C WILSON JR STADIUM":"[1] BILLS DR":"ORCHARD PARK":"NY":"14127-2237"
"Diplomatic Representation of Afghanistan in the US":"2341 Wyoming Avenue NW":"Washington":"DC":"20008"
"Diplomatic Representation of Andorra in the US: Chancery":"United Nations Plaza 25th Floor":"New York":"NY":"10017"
"General Consulate of Argentina in Houston":"1990 South Post Oak Road Suite 770":"Houston":"TX":"77056"
"Honorary Consulate of Fiji":"2050 W. 190th St. Suite 102":"Torrance":"CA":"90504"
"Honorary Consul Consulate of Finland":"1230 Peachtree Street NE Suite 3100":"Atlanta":"Georgia":"30309"
"Honorary Consulate of Ireland":"2511 NE 31st Court Lighthouse Point":"Fort Lauderdale":"FL":"33064"
"George W. Bush":" 1600 Pennsylvania Ave":" Washington":" DC":"20008"

[Squashman]

I retested Devil's new code and it took 27 minutes.

All my files do have CR/LF. I think I am going to just have to make everyone bite the bullet and use my batch file with the ported utilities. That is if they want there job to finish quickly. If they are OK with it running for a half an hour, which they may be OK with.

I was thinking of just putting all my ported utilites out on our shared drive and then just putting in the full pathnames to the commands into my batch file. That way they won't have to worry about copying over any of them to their working folder or copying them to their pc and adding the directory to their path statement.

Could I create a temporary path statement to the utilities on the network drive so that I wouldn't have to put the full path to all the commands in my batch file? That way they wouldn't have to install them onto their PC and add to their path statement.

Here is my current batch file. I added some logging into it.

@echo off
:: Time Stamp Start Time
if exist appendlog.log del /q appendlog.log
Echo exit|cmd /q /k prompt $D $T>>appendlog.log

:: comment log
echo.>>appendlog.log
echo.>>appendlog.log 
echo Input>>appendlog.log

:: Get Quantity of Input Records for each file
echo "Calculating input quantities"
for /f "tokens=2,3 delims=: " %%i in ('find /v /c "SomeStringNotToBeFound" *.CHR') do (
for  /f "tokens=1" %%l in ('set /A _NumLines^=%%j-1') do echo %%i %%l>>appendlog.log)

:: delete any existing header.txt file
if exist header.txt del /q header.txt

:: delete any existing combined_output.dat files
if exist combined_ouput.dat del /q combined_output.dat

:: create header record to reverse match against all files and output as first record to combined_output file
:: All Input files must end in the file extension CHR
head -q -n1 *.CHR | head -n1 > header.txt

:: Add Filename Field to end of header line
:: Also Adds the delimiter and surround characters around the Field name
cat header.txt | sed "s/.$/:\"Filename\"/" > combined_output.dat


:: Append the Filename to the end of each line of every file and combine all files together
:: Also adds the delimter and surround character around the variable
:: All Input Files must end in the File extension CHR
:: FOR /F "tokens=*" %%A IN ('dir /b /a-d *.CHR') DO cat %%A | grep -v -f header.txt | sed -e "s/.$/:\"%%A\"/g" >> combined_output.dat
FOR /F "tokens=*" %%A IN ('dir /b /a-d *.CHR') DO echo "Working on file: %%A" && cat %%A | grep -v -f header.txt | sed "s/.$/\":\"%%A\""/" >> combined_output.dat


:: Clean UP
del /q header.txt

:: comment log
echo.>>appendlog.log
echo Output>>appendlog.log

:: Get Quantity of Output file
echo "Calculating Output Quantity"
for /f "tokens=2,3 delims=: " %%i in ('find /v /c "SomeStringNotToBeFound" combined_output.dat') do (
for  /f "tokens=1" %%l in ('set /A _OutLines^=%%j-1') do echo %%i %%l>>appendlog.log)


:: Time Stamp finish time.
Echo exit|cmd /q /k prompt $D $T>>appendlog.log

echo "All done!!!!"
echo "Do you want to view the log file? Y/N"
set /P _Answer=

:: Open AppendLog if Answer equals Y. Not case sensitive.
if /I %_Answer%==Y appendlog.log

I am currently using the unix utils from here:
http://unxutils.sourceforge.net/

But maybe I can get everyone to install Windows Services for Unix. I could go back to my good ole days of writing shell scripts again.
http://www.microsoft.com/downloads/d...displaylang=en

[TheOutcaste]

When you run a batch file, it starts cmd.exe. All changes made to System Variables are local to that process, and are discarded when the program ends and the cmd.exe process is ended.
If you open a command prompt, run a batch file that changes the Path variable, then when the batch file ends, the change is still present, but will be discarded when the command prompt window is closed.
You can also start your file with a setlocal command. Then all variables are discarded when the file ends.
Or you can save the current path, change it as desired, then set it back as part of the cleanup. This gives a little assurance that it will work with differing versions of Windows/DOS

You can also set a variable to the path, ie set pth=\\server\share\;%path%, then call the commands using that variable, %pth%sed, %pth%head and so on

Just be sure if you change the Path variable that you include al least %systemroot%\system32;%systemroot% if not the entire current path, else Windows won't be able to find the external batch commands like find and findstr.

I re-ran my timing tests rebooting my PC before each test and all three versions took 35 minutes +/- 15 seconds. I did have a couple of backup processes that started on two of the tests, that ran for about 2 minutes, so there might be a wider variance, but it will be no-where near as fast as using the Unix utilities.
I haven't tried converting the batch file to an exe file, that might make it a bit faster, but I'm not sure how much.
This roughly matches your time taking file size into account: (562000/700000)*35 minutes = 28.1 minutes

In your script above for Calculating Output Quantity, as you are only checking the one output file (combined_output.dat) and not multiple input files, the second For loop can be removed. This should give the same result:

for /f "tokens=2,3 delims=: " %%i in ('find /v /c "SomeStringNotToBeFound" combined_output.dat') do (
  set /A _OutLines=%%j-1>nul) & Echo.%%i %_Outlines%>>appendlog.log

probably won't make more than a .02 second difference though.

Now to find my bat2exe file...

Jerry

[Squashman]

We used the bat2exe thing when I worked for the school district. Seems to be alot of different ones out on the web. I am wondering if it would be able to compile my batch file that uses my unix utilities.

[Squashman]

Thanks for reminding me about the Bat 2 exe thingy. I was able to compile my Batch script with all the ported Utilities into one Executable.

I appreciate all the help you guys have given me. I wouldn't have been able to write half the stuff I added to my batch file without looking at some of your code.

I am hoping both of you can help me with a few more things. But this one will require the use of grep but I am not sure. I think we might be able to use Findstr instead if I am correct that it can take all of its search paramters from a file.

My next order of busines is to do a profanity search. I have a file with a list of profanity words in it. One word per line. Need to search a data file and output the lines that match any of the profanity words to one file and the ones that don't match to another file. I think I could do that with a For, Type,echo and Findstr commands. Get the errorlevel of findstr and echo the line to the appropriate output file. Something like

for /f %%a in (type somefile.txt) do echo %%a | findstr /grofanitywords.txt | if errorlevel==0 echo %%a >> profanity_output.txt else echo %%a>>not_profanity.txt

Something like that. I know the syntax isn't correct. Just trying to convey what I am thinking in a more technical manner.

I should really start a new thread.

[Squashman]

Originally Posted by devil_himself

Now This Script Doesn't Care About The File Type

@echo off
setlocal enabledelayedexpansion
set tp=c:\tp.txt
if exist c:\tmpfile.txt del /q c:\tmpfile.txt
(for /f "tokens=*" %%a in ('dir /b /a-d') do (
   if not %%a==%~nx0 (
   (for /f "delims=" %%i in ('cmd /c for /f "delims=" %%j in (%%a^) do echo %%j ^^^& exit') do set f=%%i)
     (for /f "skip=1 usebackq delims=" %%b in ("%%~dpnxa") do (
     echo %%b:"%%~nxa" >> "%tp%"
    )
  ))
))

echo %f%:"Filename">c:\tmpfile.txt
type "%tp%">>c:\tmpfile.txt
del /q "%tp%"

Trying to understand the syntax of your Batch file. Could you explain a few things for me.

if not %%a==%~nx0 (You are comparing the current filename to what?)

What are these variables getting assigned:
%%~dpnxa
%%~nxa

Not quite understanding the whole Tilde thing.

[devil_himself]

To use the FOR command in a batch program, specify %%variable instead
of %variable. Variable names are case sensitive, so %i is different from %I.

Inside A Batch Script Two "%'s" in %%variable, are required .The first % escapes the second %,
so that the for command to actually work. On the command line itself, you only use one %.

Complete Modifier List - for /?

 %~I         - expands %I removing any surrounding quotes (")
    %~fI        - expands %I to a fully qualified path name
    %~dI        - expands %I to a drive letter only
    %~pI        - expands %I to a path only
    %~nI        - expands %I to a file name only
    %~xI        - expands %I to a file extension only
    %~sI        - expanded path contains short names only
    %~aI        - expands %I to file attributes of file
    %~tI        - expands %I to date/time of file
    %~zI        - expands %I to size of file
    %~$PATH:I   - searches the directories listed in the PATH
                   environment variable and expands %I to the
                   fully qualified name of the first one found.
                   If the environment variable name is not
                   defined or the file is not found by the
                   search, then this modifier expands to the
                   empty string

The modifiers can be combined to get compound results:

    %~dpI       - expands %I to a drive letter and path only
    %~nxI       - expands %I to a file name and extension only
    %~fsI       - expands %I to a full path name with short names only
    %~dp$PATH:I - searches the directories listed in the PATH
                   environment variable for %I and expands to the
                   drive letter and path of the first one found.
    %~ftzaI     - expands %I to a DIR like output line

==
1.if not %%a==%~nx0 (You are comparing the current filename to what?)

     %~nI        - expands %I to a file name only
     %~xI        - expands %I to a file extension only
     %0           - batch file

>> %~nx0 --> name and extension of batch file
>> %%~dpnxa --> drive,path,name,extension of the current file
>> %%~nxa --> name and extension of the current file
==

[TheOutcaste]

The %0 variable will give you the name of the batch file as it was called.
This is important, as %0 by itself may not include the extension or path, so you have to use the modifiers to be sure of what you are getting. Note that %0 by itself will return with the same case as you called the file.

Examples for a batch file called Test.cmd located in the c:\scripts\temp folder:
C:\scripts\temp>test
%0 will be test
C:\scripts\temp>teSt
%0 will be teSt
C:\scripts\temp>.\test
%0 will be .\test
C:\scripts\temp>..\temp\test
%0 will be ..\temp\test
C:\scripts>temp\test
%0 will be temp\test

%~nx0 will always return as Test.cmd including the case of the filename.

Jerry

[Squashman]

Just like $0 in a bash script. I was thinking that it what it was but wasn't sure. The modifier list certainly comes in handy.

[Squashman]

Can the modifier list only be used in a For statement?

[devil_himself]

Originally Posted by Squashman

Can the modifier list only be used in a For statement?

hmm .. no

@echo off
  setlocal
  if "%~1"=="" (
    echo Please specify the name of the file
    goto :eof
  )
 
  if %~x1==.pdf echo this is PDF 
  if %~x1==.xls echo this is xls

::Determine The Path That a Batch File Is Run From
Echo %~dp0

[Squashman]

I guess I should have rephrased that. You can't use the modifiers with a variable. Lets say you alredy set a variable to the full path with the filename. Can you use that variable witth the modifiers.

[devil_himself]

Originally Posted by Squashman

Lets say you alredy set a variable to the full path with the filename. Can you use that variable witth the modifiers.

If I Understand Correctly Then

::extract the last part of a path
@echo off 
setlocal
set var=C:\ABC\DEF\GHI\IJK
echo %var%
for %%a in ("%var%") do echo %%~na

[Squashman]

I figured as much. I thought it would be easier then that. I was hoping I could just do it in a set statement but I guess you can't use the modifiers with a variable you already set. I was hoping you could do it like this.
set var=C:\ABC\DEF\GHI\IJK\some.txt
set new_var=%%~dpvar