reading a csv file in powershell , header line having duplicates

Question

I have a script I am trying to read a csv file with and assign column names to the data. however the files has multiple columns named the same see below Grade QA/QC and Symbol

 ID,Date,Water Level / Niveau d'eau (m),Grade,Symbol / Symbole,QA/QC,Discharge / Débit (cms),Grade,Symbol / Symbole,QA/QC
07QC008,2018-12-28T00:00:00-06:00,6.536,,,1,0.052,,,1

here is the code below, how can I modify it to maybe strip out the header row and susbstitute my ow columns names?

$stations = import-csv “C:\Users\robm\Downloads\stations\SK_hourly_hydrometric.csv”
#$stations | Get-Member


ForEach ($station in $stations){
$ID = $($station.ID)
$Date = $($station.Date)
$WaterLevel = $($station."Water Level / Niveau d'eau (m)")
$GradeWL = $($station.Grade1)
$SymbolWL=$($station."Symbol / Symbole1")
$QAWL=$($station."QA/QC1")
#$Discharge=$($station."Discharge / Débit (cms)")
$Discharge=$($station."Discharge")
$GradeD=$($station.Grade2)
$SymbolG=$($station."Symbol / Symbole2")
$QAD=$($station."QA/QC2")
Write-host $ID "|" $Date "|" $WaterLevel "|" $GradeWL "|" $SymbolWL "|" $QAWL "|" $Discharge "|" $GradeD "|" $SymbolG "|" $QAD
#Write-host $Discharge
}

Answer 1

You can't import with the duplicate headers,
so ignore them and supply your own new header names.

• use Get-Content instead of Import-Object
• skip the first line
• use ConvertFrom-Csv with your header
• (de-)select the column(s) you don't want in the output
• use export-csv with -Delimiter '|' to have a new proper csv
  (with all headers/columns double quoted)

---

## Q:\Test\2018\12\31\sf_947091.ps1
$FileIn = 'C:\Users\robm\Downloads\stations\SK_hourly_hydrometric.csv'
$FileOut= 'C:\Users\robm\Downloads\stations\SK_hourly_hydrometric_new.csv'

$stations = Get-Content $FileIn | Select-Object -Skip 1 |
    ConvertFrom-Csv -Header ID,Date,WaterLevel,GradeWL,SymbolWL,QAWL,Discharge,GradeD,SymbolG,QAD 

$stations | Select-Object -Exclude Discharge 

$stations | Select-Object -Exclude Discharge | Export-Csv $FileOut  -NoTypeInformation

Sample output to screen:

ID         : 07QC008
Date       : 2018-12-28T00:00:00-06:00
WaterLevel : 6.536
GradeWL    :
SymbolWL   :
QAWL       : 1
Discharge  : 0.052
GradeD     :
SymbolG    :
QAD        : 1

Sample $FileOut

> Get-Content $FileOut
"ID"|"Date"|"WaterLevel"|"GradeWL"|"SymbolWL"|"QAWL"|"Discharge"|"GradeD"|"SymbolG"|"QAD"
"07QC008"|"2018-12-28T00:00:00-06:00"|"6.536"|""|""|"1"|"0.052"|""|""|"1"

---

A quite different approach is to automatically read the old headers,
store in a hash table,
thereby checking for duplicates and appending an increasing number

## Q:\Test\2018\12\31\sf_947091_".ps1
$FileIn = 'C:\Users\robm\Downloads\stations\SK_hourly_hydrometric.csv'

(Get-Content $FileIn | Select-Object -First 1) -split ',' |
ForEach-Object {$Header = @{};$i=0}{
    $key = $_
    $j = 1
    while ($Header.Containskey($key)){
      $key = ("{0}{1}" -f $_,++$j)
    }
    $Header.Add($key,++$i)
}
$NewHeader = ($Header.GetEnumerator()|Sort-Object Value|
    ForEach-Object {'"'+$_.Name+'"'}) -Join ','

$NewHeader

---

"ID","Date","Water Level / Niveau d'eau (m)","Grade","Symbol / Symbole","QA/QC","Discharge / D‚bit (cms)","Grade2","Symbol / Symbole2","QA/QC2"

Answer 2

Based on "lotpings" comment, I wrote a function that verifies if the CSV has no doubles. It also makes sure the quotes from titles are removed.

use as:

csv_header_numbering_when_doublefound("dir/myfile.csv");

function csv_header_numbering_when_doublefound($fname)
{
$file=Get-Content "$fname"
($file | Select-Object -First 1) -split ',' |
ForEach-Object {$Header = @{};$i=0}{
    $key = $_ -replace '"',''
    $j = 1
    while ($Header.Containskey($key)){
      $key = ("{0}{1}" -f $_,++$j)
    }
    $Header.Add($key,++$i)
}
$NewHeader = ($Header.GetEnumerator()|Sort-Object Value|
    ForEach-Object {'"'+$_.Name+'"'}) -Join ','


$NewHeader > $fname;
$file | select-object -skip 1 >> $fname

}

Answer 3

The most simple solution seems to me is using column numbers, but you can also replace the headers according to https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/import-csv?view=powershell-6. Last column would be

$station[10]