What is the problem with retail competition in electricity?

I don't think anyone really knows.

There have been a number of studies into the electricity industry in New Zealand but none that I am aware of have really analysed the retail market. There have been a few that have purported to, such as Wolak, but his study was really a wholesale market study (with flaws). The linkages he made to the retail market were somewhat nebulous and completely about correlation without any attempt to discuss causation.

The most comprehensive study that I have seen about the retail market in New Zealand was done by the Electricity Commission in their Market Design Review.
http://www.electricitycommission.govt.nz/pdfs/opdev/wholesale/market-design/Market-Design-Review.pdf

This review did some interesting analysis around the retail market. While this was some of the most in depth analysis that had been done on the retail market it actually lead to asking more questions than it answered. Particularly interesting of the ECs analysis was Figure 27 of chapter 2 page 28. These four charts demonstrate something quite strange. When it comes to large or high density networks (which New Zealand doesn't have many of) there appears to be a correlation between size and low retail margins (which is what you would expect with economies of scale). However, in small networks retail margin is random. Small networks can have very low retail margins, very high retail margins and everything in between. The reason for this is not clear.

Much public debate has been made of location and transmission constraints in preventing retail competition but, again, the evidence is correlation only (with no empirical analysis on causation) and is inconsistent. While generator-retailer operations do tend to be distributed around geographic areas, which can be explained by locationally restricted competition but can also be explained by the starting position of the generator-retailers and relatively low effective rates of switching. In other words correlation does not necessarily mean causation. More interestingly the concentration of generation does not line up with the margin uplift. Areas where there is significant regional generation (and therefore low location risk such as the central North Island) have networks with relatively high margins. Correspondingly, many of the small networks with low retail margins are remote from significant generation and should be expensive (if location risk is the real problem).

I don't know what the answer is but I do know that just about every review that has been done looks almost exclusively at the wholesale market. The wholesale market has been done to death, the New Zealand wholesale electricity market has been constantly reviewed and there is mountains of international literature on wholesale electricity markets. It is the retail market that needs a proper empirical review. The retail market is not an adjunct to the wholesale market. The retail market relies on the wholesale market for its input prices but in every other respect it is a different market with a different design and structure. It needs its own focus.