Live Chat & Podcast at 1:00PM Eastern on Sunday!
There's no such thing as a stupid question, but they're the easiest to answer.
JoinTour
Login
Search
Business Applications
Tag Cloud
access acer asus bios bsod computer crash desktop driver drivers error ethernet excel freeze gaming gpu hard drive hardware hdmi internet laptop malware memory monitor motherboard netgear network printer problem ram registry router security slow software sound trojan ubuntu 11.10 uninstall usb video virus vista wifi windows windows 7 windows 7 32 bit windows 7 64 bit windows xp wireless
Search
Search for:
Tech Support Guy Forums > Software & Hardware > Business Applications >
Importing multiple csv files and transposing data

Reply  
Thread Tools
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
23-Jan-2010, 08:40 PM #1
Question Importing multiple csv files and transposing data
Hi everyone – I’m new here , & have some programming skills but that was when basic was actually ‘basic’ – vba is a little bit alien at the moment unfortunately.

I’m working with a macro culled from http://forums.techguy.org/business-applications/697763-excel-importing-multiple-files.html code found at the bottom of the thread.
This provides almost what I need… So if Rolin_again is around still, I’m sure it would be a quick fix.

The basic requirement is ;
Collecting data onto one worksheet from a multiple csv files each named with a time_stamp & stored within one folder. Each file has two columns only ; A=frequency ( F1 to Fn) and B = measurement values.

The layout required on the worksheet is transposed such that row 1 has the frequency range F1 to Fn which is now the common header. Row 2 has time_stamp in first cell followed by the associated data values. Row 3 to last, is set out in similar fashion.
The macro currently gives a dialog box to select the folder, checks for correct order, then copies and pastes to a new worksheet in a vertical fashion. Unfortunately the data overlaps and the measurement values are lost.

I believe what is needed is ; the first file needs copying , transposing and pasting in. this will include the F1 to Fn header.
The next file needs only Col-B (values) copying , transposing and pasting into row 2. (Col-A will have the time_stamp) followed by the data values – the remaining files follow in a similar vein.

I’m struggling to understand the last loop ( K=1 to next K) and how best to modify this to give the result required.

If anyone can assist here I’d appreciate….
Many thanks in anticipation.

Regards,
jacksocket
Zack Barresse's Avatar
Computer Specs
Distinguished Member with 5,030 posts.
 
Join Date: Jul 2004
Location: Oregon, United States
Experience: I'ma learnin'!
23-Jan-2010, 08:54 PM #2
Hello, and welcome to the board!

So let me get this straight. File 1 you want A1:Bx (where x is the unknown last row of data) to be pasted into a target file (a new file I'm assuming) to A1:Bx. Then every subsequent file after that, do you want that data (I understand only the data from column B) into the next column to the right (i.e. col C for File 2, col D for File 3, etc)? Or are you really talking about going down rows with your data? In other words, transposing the data as you've said.

So if transposing data, File 1 data A1:Bx would go in A1:Z2 (Z column being however long your data range is, so may not be exactly the same column). Then File 2 column B data (B1:Bx) would be from A2:Z2 (again, Z being relative). Is this how you want it?

Edit: Oh, and do you want to have a folder picker so you can choose the folder for this to run on? Or choose all of the files from a folder browser? And shall it include all files in this folder to draw from?
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
24-Jan-2010, 03:56 PM #3
Red face further info / confirmation
Hello Zak,
I didn't expect to have such a quick response!! thanks.
Yes, I think you understand what is needed - looking to compile the data in rows. so probably better to transpose the data on the way in rather than do the whole sheet afterwards ( is that how you'd tackle this?).
To be sure on requirement - I have attached 4 (cut down) data files, [note; they are named with a time_stamp and always have the frequency list]
also a file showing the layout of the output (colours only shown for clarity!).
as for the file picker - the files will always be in one folder and all need to be loaded to the worksheet. I guess it would be nice to re-name the output file the same as the source folder....
hope this make sense... many thanks for looking at this for me .

ps any notes in the code will be useful ..

jacksocket..
Attached Files
File Type: zip example_files.zip (4.3 KB, 11 views)
Zack Barresse's Avatar
Computer Specs
Distinguished Member with 5,030 posts.
 
Join Date: Jul 2004
Location: Oregon, United States
Experience: I'ma learnin'!
24-Jan-2010, 07:04 PM #4
Okay, if I understood you correctly, the following code should work. I made three separate modules to place this code into (it helps me to keep them separated by functionality)...

In a module named modFunctions:
Code:
Option Explicit

Public Sub TOGGLEEVENTS(blnState As Boolean)
'Originally written by Zack Barresse
    With Application
        .DisplayAlerts = blnState
        .EnableEvents = blnState
        .ScreenUpdating = blnState
        If blnState Then .CutCopyMode = False
        If blnState Then .StatusBar = False
    End With
End Sub

Public Function ISWBOPEN(wbName As String) As Boolean
'Originally found written by Jake Marx
    On Error Resume Next
    ISWBOPEN = Len(Workbooks(wbName).Name)
End Function
In a module named modFolder:
Code:
Option Explicit

'####################################################################
'####################################################################

'  Routine used from VBA Express, Knowledge Base entry by Ken Puls:
'  http://www.vbaexpress.com/kb/getarticle.php?kb_id=284

'####################################################################
'####################################################################

Function BrowseForFolder(Optional OpenAt As Variant) As Variant
'Function purpose:  To Browser for a user selected folder.
'If the "OpenAt" path is provided, open the browser at that directory
'NOTE:  If invalid, it will open at the Desktop level

    Dim ShellApp As Object

    'Create a file browser window at the default folder
    Set ShellApp = CreateObject("Shell.Application"). _
                   BrowseForFolder(0, "Please choose a folder", 0, OpenAt)

    'Set the folder to that selected.  (On error in case cancelled)
    On Error Resume Next
    BrowseForFolder = ShellApp.self.Path
    On Error GoTo 0

    'Destroy the Shell Application
    Set ShellApp = Nothing

    'Check for invalid or non-entries and send to the Invalid error
    'handler if found
    'Valid selections can begin L: (where L is a letter) or
    '\\ (as in \\servername\sharename.  All others are invalid
    Select Case Mid(BrowseForFolder, 2, 1)
    Case Is = ":"
        If Left(BrowseForFolder, 1) = ":" Then GoTo Invalid
    Case Is = "\"
        If Not Left(BrowseForFolder, 1) = "\" Then GoTo Invalid
    Case Else
        GoTo Invalid
    End Select

    Exit Function

Invalid:
    'If it was determined that the selection was invalid, set to False
    BrowseForFolder = False

End Function
In a module named modData:
Code:
Option Explicit

'Set a public constant variable
Public Const DNL As String = vbNewLine & vbNewLine

Sub ImportData()

    'Declare all variables
    Dim wb As Workbook, ws As Worksheet
    Dim wbX As Workbook, wsX As Worksheet
    Dim i As Long, iRow As Long, iFileNum As Long, sMsg As String
    Dim vFolder As Variant, sSubFolder As String, sFileName As String
    Dim bOpen As Boolean
    
    'Turn off some application-level events to improve code efficiency
    Call TOGGLEEVENTS(False)
    
    'Have the user choose the folder
    vFolder = BrowseForFolder()
    
    'Exit if nothing was chosen, variable will be False
    If vFolder = False Then Exit Sub
    
    'Check if this is what the user wants to do, confirm with a message box, exit if no
    sMsg = "Are you sure you want to import data from this folder:"
    sMsg = sMsg & DNL & vFolder
    If MsgBox(sMsg, vbYesNo + vbDefaultButton2, "ARE YOU SURE?") <> vbYes Then Exit Sub
    
    'Set sub-folder as variable for save name at end of routine
    sSubFolder = Right(vFolder, Len(vFolder) - InStrRev(vFolder, Application.PathSeparator))
    
    'Set destination file with one worksheet
    Set wb = Workbooks.Add(xlWBATWorksheet)
    Set ws = wb.Sheets(1)
    
    'This will be the row to start data on, to incriment in loop
    iRow = 2
    
    'Loop through files in folder
    sFileName = Dir$(vFolder & "\")
    Do Until sFileName = ""
    
        'Check that the file pattern matches what you want, i.e. 12.16.00.xls
        If sFileName Like "??.??.??.xls" Then '### set file extension here
        
            'Check to see if the file is open
            'If file is open, set as variable, if not, open and set as variable
            If ISWBOPEN(sFileName) = True Then
                Set wbX = Workbooks(sFileName)
                bOpen = True
            Else
                Set wbX = Workbooks.Open(vFolder & "\" & sFileName)
                bOpen = False
            End If
            
            'Set first sheet in target workbok as worksheet variable, from which to mine data
            Set wsX = wbX.Sheets(1)
            
            'Get last row from column A (range for copy/pasting)
            i = wsX.Cells(wsX.Rows.Count, 1).End(xlUp).Row
            
            'Check if a file has been added, if not add headers (frequency)
            If iFileNum = 0 Then
                ws.Range("B1", ws.Cells(1, i + 1)).Value = Application.Transpose(wsX.Range("A1:A" & i))
            End If
            
            'Add data
            ws.Range("B" & iRow, ws.Cells(iRow, i + 1)).Value = Application.Transpose(wsX.Range("B1:B" & i))
            
            'Add file name to column A
            ws.Range("A" & iRow).Value = "'" & Left$(sFileName, Len(sFileName) - 4)
            
            'Incriment variable values
            iRow = iRow + 1
            iFileNum = iFileNum + 1
            
            'If file was closed to start with, clean up and close it
            If bOpen = False Then wbX.Close SaveChanges:=False
            
        End If
        
        'Get next file name
        sFileName = Dir$()
    Loop

    'Check if file name to save exists
    If Dir$(vFolder & "\" & sSubFolder & ".xls", vbNormal) = "" Then
        wb.SaveAs vFolder & "\" & sSubFolder & ".xls"
        MsgBox "Complete!", vbOKOnly
    Else
        MsgBox "File already exists!  File is NOT saved!", vbInformation, "COMPLETE!"
    End If
    
    'Reset events back to application defaults
    Call TOGGLEEVENTS(True)
    
End Sub
It works for me in testing. The file is saved to the folder the user chooses to run (wasn't sure if that's what you were looking for). Also, you said CSV files in your original post, but you posted XLS files, so the routine is looking for XLS files and not CSV files. If you need to change what files to look for, there's a place in the code that shows what the naming convention should look like and it would need to be changed to reflect as such.

Let us know if this is what you're looking for. Be sure to make a backup copy of your work and save everything before you run this. It doesn't affect any new files, only creates a new file. And no formatting is done to the file either.

HTH
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
26-Jan-2010, 04:18 AM #5
Zak, this looks great.... I haven't tested with a real file - but it certainly works with the test files, so I imagine it will work superbly. I have another busy day today - but will probably get a chance later this evening to put it to work for real.
I'll keep you posted & again many thanks for your efforts with this !
jacksocket..
Zack Barresse's Avatar
Computer Specs
Distinguished Member with 5,030 posts.
 
Join Date: Jul 2004
Location: Oregon, United States
Experience: I'ma learnin'!
26-Jan-2010, 09:59 PM #6
Great. As with everything, I caution about using files which aren't your "real" files on boards, or at least not describing any differences between test files and the one's you will actually be working with. (Just as a blanket precautionary note.)

Definitely let us know how it goes.
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
27-Jan-2010, 07:19 PM #7
Arrow More complications - sorry!
Hi Zak,
you were right to lower my expectations with the 'real' files...!!.
The code now throws up an error in the copy /paste routine..
My colleague had described the file layout over the phone, hence my test files were hand made ( F1 to Fx etc and values copied from similar files we use).
I have now received an actual output for the spectrum analyzer and a number of items are subtlety different, although probably fixable.
I have attached a handful of files for you to examine - :
I guess there is some incompatibility with data type??
The freq list is in numerical frequency (MHz) not channel numbers (F1 etc) also the measurement value in Col_B has more decimal places than I expected ( however this could be rounded up to be an integer).
The other most obvious difference is the 34 row header! -
This I have worked out... using the original test data . By adjusting the start of 'add headers' to ; Transpose(wsX.Range("A35:A" & i)) & 'add data' to; Transpose(wsX.Range("B35:B" & i)). This omits the header and goes straight to the data.
(just a thought.. - this info is common to all the files and might be useful to note settings etc. could this be copied into a new worksheet in the output file??)

I have made the change to locate .csv files only - will this change anything else??

apologies for the false start here but I was keen to find a solution to enable the use of data, rather than view endless (usually 30,000ish) screen shots in .pcx format which is the alternative output from the spectrum analyzer.
hope all this make sense - and you are able to advise on the way forward.
thanks again.. and sorry for any inconvenience this has caused!
jacksocket.
Attached Files
File Type: zip Real-files.ZIP (19.7 KB, 8 views)
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
02-Feb-2010, 04:21 AM #8
Hi Zak, are you able to give any further help with this ?- should i now persue some csv to xls converter code , any advice would be helpful.
thanks
jacksocket
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
02-Feb-2010, 12:20 PM #9
lets try - bump
Zack Barresse's Avatar
Computer Specs
Distinguished Member with 5,030 posts.
 
Join Date: Jul 2004
Location: Oregon, United States
Experience: I'ma learnin'!
02-Feb-2010, 04:04 PM #10
Haven't had time to look at this. Will try to today.
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
03-Feb-2010, 04:17 AM #11
you're a star! thanks -
i only found the bump feature yesterday.
looking forward to your reply...
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
09-Feb-2010, 04:34 AM #12
can anyone help with this .....? I cant get the csv's to load - not sure whats stopping it...any ideas..
Rollin_Again's Avatar
Senior Member with 4,273 posts.
 
Join Date: Sep 2003
Location: Atlanta, GA - Planet Earth
Experience: Brilliant When Sober
10-Feb-2010, 04:48 PM #13
Are you getting an error when you run the code?

I also assume you change the line of code below so that it is looking for CSV files instead of XLS?

Code:
If sFileName Like "??.??.??.CSV" Then '### set file extension here
Rollin
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
11-Feb-2010, 04:59 AM #14
Hello Rollin,
I spent some time on the code the other night - trial and error , finding what make a chages etc...
I seems the data format doesn't make any difference - I opened my test files , pasted in the real data . it collects it sucessfully and gives the correct output. no worries there..
Also I've worked out the reason why I was getting blank worksheets on occasion.... the files I was pointing at had file etxn. in caps as '... .CSV ' aghh.. Additionally the file title on some were other than xx,xx,xx.csv ie longer names on the real files - my coleague had included a site loaction too! So now I changed the code to include '*' to pick up any files.
Also to get around the 34 row header on each file - the code now looks like...
for the check for header ...
ws.Range ("B1", ws.Cells(1, i-34). Value = Aplication.Transpose(wsX.Range("A35:A" & i))
in the add data line....
ws.Range ("B"& iRow, ws.Cells(iRow, i -34). Value = Aplication.Transpose(wsX.Range("B35:B" & i))
This omits the top 35 rows - however I think a line of code would be good to collect it during the >>
If iFileNum = 0 ( header checking line) to copy it and paste it into a new worksheet ( same outut file) as a refference of the equipment settings. I haven't tried this yet - but think it would be straightforward.
However for some reason it will only pick up and display ( as a shortened version) the test files.???
I does show an error when it hits the " check file header" line on the real files - and thats the current problem . I was thinking the only difference now is the file lengh ie number of rows .
Any help would be useful on this .
Many thanks for responding to my call for help!!

jacksocket.
jacksocket's Avatar
Junior Member with 10 posts.
 
Join Date: Jan 2010
Experience: Beginner
18-Feb-2010, 04:46 AM #15
further progress made
I've done further work on this ... for those folks out there interested, i'll explain..

The reason for the error message was due to the amount of data. The test data had only a limited number of rows and the 'transpose ' was no problem - however the real data has 301 rows and of course Excel 2003 only handles up to 256 rows. The fix was to open the file - change out of 'compatibility mode' ( using MS symbol [top left] selecting 'options' / 'save' / 'save files as' and select 2007 type ' .xlsx '. and then re-save.
This is now able to cope with 16,384 rows !
I made two versions of the project one which picks up and saves in .xlsx - another which saves as *.xls ( old style) - as I'm not sure if the next stage will import the 2007 version, either way when you use the old file type just select 'yes' when it tells you "this file is in a different format to the specified file extension" etc. when you open the resultant file.

The next chalenge is to collect the header info form the first file and save into a new worksheet on the target file ( just for reference to equipment set up etc).
I had a go, but ended up aborting to a good working version - glad I made a copy before corpting the code!
If anyone has done this - please post me a link ; or advice from the guys that know this stuff inside-out.
I've attached my current working copy with a small number of test files.

cheers - jacksocket
Attached Files
File Type: zip DataCompiler - saves all cols as xls.zip (31.0 KB, 10 views)
Reply

Tags
excel, vba, windowsxp

THIS THREAD HAS EXPIRED.
Are you having the same problem? We have volunteers ready to answer your question, but first you'll have to join for free. Need help getting started? Check out our Welcome Guide.

Search Tech Support Guy

Find the solution to your
computer problem!




Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
WELCOME TO TECH SUPPORT GUY! Are you looking for the solution to your computer problem? Join our site today to ask your question -- for free! Our site is run completely by volunteers who want to help you solve your computer problems. See our Welcome Guide to get started.
Thread Tools



Facebook Facebook Twitter Twitter TechGuy.tv TechGuy.tv Mobile TSG Mobile
You Are Using:
Server ID
Advertisements do not imply our endorsement of that product or service.
All times are GMT -4. The time now is 02:10 AM.
Copyright © 1996 - 2011 TechGuy, Inc. All rights reserved.

Powered by Cermak Technologies, Inc.