[Jimmy the Hand]

Hello World,

I'm not sure if this is the proper forum to ask, but I don't know of a better place. Okay, here's the task.

I want to reproduce a UV spectra plot from the raw data that a UV spectrophotometer provides. The instrument measures a Signal value and a Background value. From these two, with a correction Factor, it calculates a Corrected Signal.

The spectrometer controlling software can export raw and manipulated signal data to text file. So, from a certain laboratory test I exported the corresponding Signal, Background and Corrected Signal values as function of time. See attached Excel table.

What I'm looking for is how the Corrected Signal is calculated. In theory (according to user manuals), the formula is

Corrected Signal = Signal - Factor*Background

where Factor is a constant numeric value. In practice, however, this formula doesn't work. But the real formula must be something similar.

So I'm looking for a function, an algorithm that calculates exactly the given Corrected Signals from the given Signal and Background values.

Can anyone tell me how to start with this? Any suggestions, advices would be highly appreciated.
Thank you in advance,

Jimmy

[OBP]

Jimmy, sorry but your equation does not add up at all, it is out by a magnitude of 100.
It is much closer with C2/b2*i1 but even then you don't get an actual constant that gives all the results that you have.
The constant that comes closest is 1.164846039

See this for Signal to Noise Ratios.
http://en.wikipedia.org/wiki/Signal-to-noise_ratio

[Jimmy the Hand]

Hi OBP,

thanks for replying. I'm sorry, I don't follow the first part of your post.
You see, my English was developed on fantasy books and RPGs, and still have problems with technical terms. Could you please rephrase what you wanted to say?

As for the signal to noise ratio. It's interesting but I don't think I have anything to do with it.
In the data I posted last time, Signal and Background values are real, physical, measured values. (They are, in fact, electrical charges, because the instrument is a diode array detector.) These values are influenced by noise, all right. But the Corrected Signal is a computed value, and I'm looking for the exact formula the software uses to generate Corrected Signal from Signal and Background. I don't care about noise here. I could just as well ask for a formula to generate column D from columns B and C, and never even mention instruments or signals or measured values.

I thought, as I said last time, the formula was basically
Corrected Signal = Signal - Factor * Background.

I still think this makes sense, but there must be a trick in the way the formula is used. Maybe the Factor is not a constant, but somehow generated from time or something. I was actually asking how to determine a certain (existing) correlation (i.e. the formula) between sets of data.

[OBP]

Jimmy that is what I tried to do for you, your formula if you apply it as written gives corrected values 100 time larger than the ones that you get from the computer.
So I tried figuring out a formula that would give the values that are provided by the computer.
It came out as Signal/Background*Factor where the factor is 1.164846039.
But even with that formula and factor the computed values can't be matched exactly to more than a couple of decimal places.
I thought the signal to noise ratio is what you are doing, i.e. the background is the "noise" and the factor is the "ratio".

[IMM]

Something pretty fishy when background>signal unless signal has already had background removed from it.
Is background in different units than signal ( eg. background is in micro.. and signal in milli... ?)

--------
I'm not going to try and sort it out - but if you want a small treatise on the type of thing that is normally done - try this
http://www.varianinc.com/cgi-bin/reg.../uv77.pdf'

[OBP]

That is good thinking IMM.

[IMM]

Actually - I suspect the numbers are raw I and Io values and he has to do the std. transmittance and absorbance thing with it.
If that is 2 channel data - it's not really surprising - but I dislike calling the reference channel "background".
I'd guess his "factor" will represent some offset between 2 beams or channels -but it would be nice to know that the detector(s) are linear and don't follow something like a diode equation.
If the truth be told tho' -- I am really too lazy to really look at it w/o knowing the physical setup.

Maybe he'll come back to this thread and I wont have to wonder if this is a dual beam kinetic study done at a fixed wavelength - or if it's a wall street stock chart

[Jimmy the Hand]

First of all, I thank you very much for trying to help me. I didn't want to go into details, because my case is very special, so it requires a lot of explanation, and I didn't see much hope that anyone can tell me exactly the thing I want. But maybe it's worth making the effort, after all.

So.
I said at first that this was a UV spectrometer. Well, not exactly. Actually, this is an Atomic Emission Detector for Gas Chromatography. The principles of operation are the following. Compounds eluting from a GC column are carried (by GC carrier gas) into a microwave induced plasma, where they are atomized and the atoms excited. The excited atoms emit characteristic light, which goes into the spectrometer part. Here there is a rotating grating that resolves the incoming light according to wavelength, and sends the light to a photo diode array (PDA). By rotating the grating, the instrument controls the wavelength range that is detected by PDA, e.g. 170-196 nm, 480-501 nm, or any possible slice of the whole spectra.

This detector is used for selective detection of heteroelements like Sulfur, Nitrogen, Oxygen, etc. My reason to mark it as a spectrometer is that AED is a rare type of instrument. For example, in our country there are only 4 pieces. Chemists know spectrometers much better, as it is 10 times more widespread. Also, AED is, in fact, a spectrometer. The paricular application I'm trying to make is about analysis of Sulfur. Sulfur is best detected around 181 nm. This (the 181 nm) is the reason why I marked my instrument as a UV spectrometer. It might have been misleading, sorry. This instrument measurers emission, not absorbance. There's no reference beam here, nor lamp intensity fluctuations, cuvette or solvent absorptions. There's only intensity of the emitted light. And noise, and background.

Now, about signal generation.
The detector is sensitive of sulfur but, in the vast majority of cases, sulfur is not alone in the plasma. Almost always ther's carbon there as well. When the original sample is petroleum related, as is my case, it's only natural that you can't, by gas chromathography, separate a sulfur compound from the hydrocarbon matrix, so the palsma will always be full of carbon, and there might also be a little sulfur present. The signal of the carbon is what I call background. It' not the same as noise. See the attached images. "Carbon background.jpg" shows the effect of carbon on spectra. In H2S there is no carbon, so it's a beautiful sulfur spectra. In hexil-mercaptan (C6H14S) there are 6 carbon atoms, so backgroung is apparent, and if hexil-mercaptan is dissolved in gasoline, the background is much higher, because of the hydrocarbon matrix. In real samples, carbon background can be significantly higher than sulfur signal.

"Signal Generation.jpg" shows the metod of background correction of sulfur signal. In the diode array, some diodes are designated as sulfur signals, others are designated as backgrounds. Signal diodes are combined to one value, background diodes are combined to another value. As the figure shows, Net (or Corrected) Signal is obtained by subtracting Backgroung value from Signal value, and here is the mysterious BackAmount, which I called factor previously.

The instrument controlling software is a modified version of Agilent Chemstation v10.02. It stores signal and background in two separate sets of raw data. (They are sets of data, because signals are recorded as function of time, that's what makes it a chromatogram.) Chemstation allows me to load signal and background chromatograms separately, and export them into csv files. If I load signal and background chromatograms together, Chemstation does the subtracting, and in the meantime it calculates an ideal BackAmount (factor), which can be different for every single sample. I can export the corrected signal, as well, to csv. That's how I produced the 3 sets of data I posted before.

Now, my task is to write a software to expand the capabilities of Chemstation, because we want to do a new type of sample evaluation. I've already decoded Agilent binary (raw) data fies, so I can convert them into series of numbers vs time. But I'm stuck with this background correction thing. I can't calcuate my final result from Signal only, because that would definitely be wrong, I mean, very wrong. I must have the method Chemstation uses to generate a corrected signal.

After all this explanation, I think it's not important anymore, how I got the 3 series of data. I just need one formula that calculates one from the other two. Or, I need an algorithm that allows me to find this formula for myself.

[OBP]

IMM, you were unbelievably close with analysis.
Jimmy, this says it all (quote)
If I load signal and background chromatograms together, Chemstation does the subtracting, and in the meantime it calculates an ideal BackAmount (factor), which can be different for every single sample.
Can you just work it in reverse to obtain the Backamount (factor) that the Machine has used, won't that be enough for your analysis?
I don't think you are ever going to be able to calculate it yourself from the raw data as it obviously uses an algorithm you can't get access to, unless you can get it from the Machines makers. The signals may be peak values or they may be "area under the curve" values or rms values.

[Jimmy the Hand]

Hi, OBP,

Knowing the BackAmount is no help at all. You see, I know the BackAmount, because Chemstation tells me, if I ask. The problem is that, if I apply the given formula to calculate Corrected Signal values, and use the BackAmount that Chemstation gives me, I get values different from those that are calculated by Chemstation. This means that either the formula is wrong, or the BackAmount (given by Chemstation) is wrong, or there is some hidden trick in the calculation.

You have analyzed my sets of data and said that there is not a single constant BackAmount that's valid for all Signal/Background/Corrected Signal trios. On the other hand, the formula was given to me on an official AED training, by official AED staff, in Atlanta, 1997. It must be correct, or else Agilent (or HP, I think back then it was still Hewlett-Packard) people were lying. I don't want to accept this possibility, so I'm more inclined to suspect that there is a trick in the calculation. All the more so, because I've had a tricky issue with HP/Agilent raw data format in the past.

The story is: Chemstation has always had the functionality to display raw data or export it into textfile. But the displayed/exported values were a constant 4/3 times greater than those stored in the binary file. And they were rounded, to spice the problem a bit more. This little trick caused me not a small headache when I was trying to unveil the secrets of the binary data structure.

So, my suspect is that the formula will have to be modified. Like if Chemstation gives me a BackAmount of 1.0168, it is really 0.0168, or Exp(1.0168) or something.

By the way, signals are electronically generated this way (or so I was told):
There are the photodiodes in the array. Each of them are periodically (e.g. 5 times per seconds) charged up to full capacity. When they are contacted by light, they lose some electric charge. Next time they are recharged, the instrument measures the amount of charge that is needed to reach full capacity again. So this measured charge is an avarage value over the recharge period, and is proportional to the amount of light emitting atoms going through the plasma during a recharge period.

[IMM]

AED is a rare type of instrument

I'll say -- never got much more exotic than Ni63 ECD, hot salt NPDs or flame photometric myself -- but I'll go through what you wrote and give it a little thought.
Have to sleep now tho'

As an aside - organo-sulphur labs out to be on remote islands.
Most peoples noses aren't very fond of them

[OBP]

Jimmy, when you ask the Chemstation for the "factor", are they all different values?
If so can you post some so that we can see ant possible correlation?

[Jimmy the Hand]

Originally Posted by OBP

Jimmy, when you ask the Chemstation for the "factor", are they all different values?
If so can you post some so that we can see ant possible correlation?

I'm not sure I understand.
When I load the signal and background chromatograms of a sample, as a result, I see only one chromatogram, which is calculated from the loaded ones by Chemstation. It uses uses an unknown algorithm to determine an auto-calculated BackAmount factor, which is best suitable to eliminate matrix interference from that particular chromatogram. Auto-calculated factors depend greatly on the composition of their respective samples, so yes, they are practically all different.

In the software there is a menu where I can choose from recommended BackAmount factors, or supply one myself. There is a default factor, which is determined by analysing a standard test sample, there is the auto-calculated one, which differs from sample to sample, and there is a user supplied value. The auto-calculated factors are most of the time OK. But if I don't like what I see, I can manipulate the chromatogram by supplying any factor I want to the background correction procedure. Then Chemstation recalculates the chromatogram and displays the new one. Changing the BackAmount factor has a significal effect on the chromatogram.

But there is only one single factor for each chromatogram. Any choosen (either recommended or user supplied) factor value is applied for the entire chromatogram, each datapoint of it.

So, I can give you any factor value you want, but it makes sense only if I provide the respective calculated chromatogram data points as well. Is this what you'd like? If so, do you want specific factors, or any one will do?

[OBP]

Jimmy, that is right, I would like some results that you are trying to evaluate plus the "factor" that goes with each set.

[Jimmy the Hand]

Hi OBP and IMM,

I exported the Background and Signal data, and Corrected Signals with 6 different Factor values, so that I can supply you with enough data to proceed. Somewhere during the process I noticed that the Signal data is not the same as the one I posted earlier. I swear I followed the same procedure of exporting raw data as before, yet I found that I can't reproduce that particular data column. I can't for life tell how it happened. Seems like I have messed up something.

Anyway, with the new Signal data the formula works like charm, as it should, so I guess this whole thread was a good big chase after shadows.
Sorry for stealing your time, and thanks anyway.

Regards,
Jimmy

Edit:
For no obvious reason I decided to upload the new data columns. May someone have fun with it.