Nearly, Peter, but not quite. Peter's second and fourth points are the key ones here:
2) "the revenues...[from green taxes]...are used to fund general expenditure"
4) "every £1 of tax needs to be compared not against the environmental cost of, say, the carbon emitted by the taxed activity, but what would have been the environmental cost of all the carbon not emitted due to the influence of the tax on consumer behaviour"
His second point is that made by the Treasury, and he's right. It is only the extent to which these taxes are higher for a green purpose that their cost should be compared to the environmental externality.
Peter's fourth point, though, is not quite right - though he will like the correct answer.
The correct answer is that the economic ideal of an environmental tax (described by Pigou long, long ago) is that the extra cost that tax imposes on the marginal consumer should match the extra otherwise-ignored environmental cost of consumption at the margin. So, if my using extra petrol of amount X imposes environmental cost Y, then the green element of petrol taxation should impose an extra cost Y upon me for consuming my last X of petrol. So far, this sounds like what the TPA measured, but it isn't. Because if marginal environmental costs rise as we use more and more petrol - so, perhaps the environmental damage of the first mile is only £0.01, but of the last mile is £1 - then the marginal environmental cost of an extra petrol amount X will be greater than the average environmental cost of each amount X has been up to this point. Consequently, if we believe that the damage done by each extra mile driven (say) rises with each mile then if the environmental tax is set correctly to match the marginal environmental cost, it will be the case that the value of the total environmental damage is much less than the total tax imposed, which is what the TPA found.
I'll be very happy to be corrected by Matthew and withdraw the point if I have missed something about his methodology (as I have only had time to take a once-over read of Section 2, and haven't tried to work through the costings myself), but on my reading the TPA analysis thus implicitly assumes either (a) that either the environmental cost rises only proportionately to the amount of environment-damaging activites; or (b) that the economic cost of extra taxes (distortions etc.) also rises more-than-proportionately with those taxes, at a rate matching or exceeding that of the environmental cost.
As it happens, I don't think it is at all obvious whether the marginal environmental cost is accelerating or decelerating, so if the TPA is assuming proportionate rises that wouldn't be silly, but if I were advising the government in how to defend itself, I would recommend them to urge that marginal environmental costs are indeed rising, and that these rises are faster than any marginal rises in the distortions created by taxes. Hence the TPA result would be unsurprising in itself, and compatible with green taxes being, nonetheless, Pigovian. (I emphasize, again, that I have not attempted to get right to the bottom of this point, and I would be very happy to withdraw it if Matthew wants to correct my interpretation of his methodology.)