Home

This is a (very) rare work-related entry. I mostly work on the compiler for a programming language named “Go”, and one of the problems we face is if and how we should add “generics” to a future version of Go. I can’t possibly summarize easily for a non-technical reader, but the TLDR version is (1) lots of other languages have generics (2) we’re pretty sure they’re useful (3) but they come with associated costs and complexity and we’re not sure they’re worth it. Also, “generics” is not just a single thing, there’s several semantic variants and several ways to implement them (for example, erased versus dictionary-passing versus template-stamping). So our team is collecting example stories of how Go generics would be useful in various situations — the jargon term for this is “use case”. Here’s mine:


The Go compiler uses maps fairly often, but map iteration order changes from one program execution to the next (and there are good reasons for this). It is a requirement that the Go compiler always generate the same output given the same program input, which means that any time the iteration order for a map would affect program output, we can’t directly use it.

The way we usually deal with this is to also build a slice of the elements in the map, where each time a new element is inserted in the map (each time insertion changes its size), that element is also appended to the slice, thus giving us an insertion-ordered map. This makes the code slightly clunkier and also adds a speed bump for new work on the compiler (and especially for new contributors to the compiler), where new code is first expressed plainly in idiomatic Go, but the idiomatic Go lacks a constant order, and then the order is cleaned up in a second pass.

We could also solve our problem with a datastructure handling interface{} elements, but this would add a bit of storage overhead (storing 2-element interfaces instead of 1-element pointers) and a little bit of time overhead, and our users are, for better or worse, remarkably sensitive to the time it takes to compile their programs.

With generics (either templated or dictionary-passing; templated would provide the best performance for us) we could define, say, OrderedMap[[T,U]], that would take care of all the order bookkeepping, be space efficient, and fast. If we could arrange for some “sensible” overloading, the difference between idiomatic Go and the correct Go for compiler implementation would be only the declaration of the data structure itself. I think that the overloading has to be part of the use case; if we have to use different syntax for something that is map-like, and thus could not update a program by updating just the declaration/allocation, that’s not as good.

By sensible overloading, I mean that we could define a generic interface type, say MapLike[[T,U]] with methods Get(T)U, GetOk(T)U,bool, Put(T,U), Delete(T), and Len()int, and any data type that supplied those methods could be manipulated using existing Go map syntax.

Iteration methods are also required, but trickier; a fully general solution would use an iteration state with its own methods, or we could declare that iteration state fits in an integer, which constrains the possible map implementations (though would be a very Go thing to do). The integer version could use three methods on the data structure itself, say, FirstOk(int)int,bool, NextOk(int)int,bool, and GetAt(int)T,U. The you-get-an-integer approach is a little interesting because it allows recursive subdivision as well as iteration, if NextOk works properly given an input that is not itself a valid index.

The iterator-state version could instead use the map method Begin()Iterator[[T,U]] and iterator methods Iterator[[T,U]].Nonempty()bool, Iterator[[T,U]].Next(), Iterator[[T,U]].Key()T, Iterator[[T,U]].Value()U. This doesn’t by itself support recursive subdivision; that would require more methods, perhaps one to split the iterator into a pair of iterators at a point and another to report the length of the iterator itself.

Twitter algorithms

July 2, 2017

These are my rules for making Twitter more useful.

My goal, on Twitter, is a combination of finding fun and interesting stuff and to expose myself to (certain) other points of view. At work we have training on bias, unconscious and otherwise, and on techniques for reducing it and countering it. One of the instructors mentioned that you can’t just wish unconscious bias away; apparently repeated exposure to normalizing examples is required, but it takes time (this is yet another disturbing/annoying way that our brains resemble neural nets for machine learning; in this light, unconscious bias is just the result of a lifelong biased training set.)

So as a rule, by default, if I see a post from an interesting woman, interesting PoC, interesting LGBTQIA person, I try to be a little more receptive to pushing the follow button. Lately I’ve decided, if it’s someone from another country I don’t necessarily hear from, that ought to count, too.

My subject bias is bikes/transit/housing, tech-especially-security, Boston area, Florida, liberal politics, science, cute animals.

But everywhere you go, especially politics and often science, you find trolls. I can’t even tell if they’re really people, and there’s a lot of them. I won’t learn anything from them, they won’t learn anything from me, it’s annoying to see someone wrong on the internet and not reply, but that’s a total waste of time. I tried blocktogether.org and that worked pretty well once I had imported a couple of lists, but then I heard mention of something called “blockchain”, not the distributed ledger algorithm, but instead a Chrome extension for bulk blocking.

So now, if I’m reading replies to an interesting tweet and I see some especially trolly comment, I visit the troll’s profile, and if it also looks especially trolly, then I select their followers. If I see that several other people I follow also follow the troll, maybe I stop there, I scan a few of the followers to see if they also look slightly troll-aligned (and remember, I’m not sure if these are real people or networks of bots) and if they are, then I click the “Run Block Chain” button and wait. For someone with more than about 10,000 followers, this will eventually error out for some reason, but it does add the ones that it scanned before the error. Twitter block chain is open source so I have a prayer of figuring the bug out if I really cared and fixing it in my copious free time but for now it works well enough and few trolls have that many followers.

Block chain will not block someone you’re already following, but inevitably you’ll pick up someone who you’d follow if you knew about them (@soledadobrien follows 338k accounts, including quite a few trolls). Sooner or later you’ll notice someone you’re following approvingly quote-tweeting someone you’ve blocked (this doesn’t happen that often, but it happens) and when that happens, I look through the block to see who it is, maybe unblock them, maybe follow them (this morning it was @deborahblum).
I’m a little nervous that I’m blocking lots of people I might otherwise follow if I knew about them, but after passing 100k blocked accounts the troll chatter is vastly reduced and that’s a real improvement.

One amusing side-effect is that this method bootstraps itself; once you accumulate a few troll-followers in your block list, you’ll find that any new troll’s followers include quite a few that you’ve already blocked, right now around 50% for me. You can use this to quickly sanity-check whether someone you think might be a troll is likely to be one; if a scan of their followers shows a lot of already-blocked accounts, perhaps the rest are worth blocking as well.

It would be lovely/interesting to do something more nuanced — for example, @deborahblum has 17 followers that I “know”, @soledadobrien has 48 followers I know, that could be a rule for not blocking someone in a followers list. It would be interesting to see how many people on my existing blocklist have more than N “discriminating” (not @soledadobrien) followers that I know, maybe review/unblock/follow some of them. (This smells like a sort of 2-sided pagerank to me.)

Someone might ask “why block, why not mute”? I don’t want to see these people, and I don’t want them to see me. There are other people who are actually harrassed on the internet by networks of trolls; I think this is one way to blunt the effectiveness of those networks.

I use the mute button when someone that I’m following goes off on some tedious unrelated tear and I just don’t want to hear about it for a while. It would be nice if muting had a built-in time limit.

In order to reduce my risk of being phished or otherwise have my account(s) hacked, I decided to get a Yubikey for my personal laptop. We use them at work, work is paranoid, if it’s good enough for them it is good enough for me.

So I bought a Nano4 and a NEO. I want the Nano4 so that it just stays plugged in by default, and the NEO has NFC so it can talk to my phone and tablet if I am authenticating from there. Both of these come with way more capabilities than I need, but they are the only two keys that can stay plugged in and support NFC.

Those way more capabilities are a problem because they’re all enabled by default. One authentication method that I’m not using is OTP — long press on the Nano4, and it burps a string of characters as if I had typed them. “Long Press” is what you get if your hand rests over your USB port, if it touches your leg, etc. Yubikey made a mistake here; almost certainly, the new customers for this gadget will be people less technical than me, and (more alarming) less technical than the tech support guys at work who hand these out (preconfigured) for our use, who also didn’t have an immediate answer to this question. Nobody’s going to want OTP configured, it’s incredibly annoying.

But, here you are, like me, you have your shiny new Nano4 and it does this annoying thing. How to fix? You need the YubiKey NEO Manager.

It has a window that looks like this, when you start it with your new YubiKey (NEO or Nano4) the “Change connection mode” button will include “OTP”. It’s pretty obvious from here: click the Change button, deselect OTP from the options presented, follow instructions (the key has to be removed and inserted) and you will be done and none of the U2F associations you’ve already made will be bothered (i.e., it will work exactly the same except that the annoying OTP typing burps will be gone).
YubiKeyNeoManager

If you don’t need NFC, or don’t need the key always resident in the laptop (that may have been a mistake; we use them several times daily at work, but I don’t for my personal account) you can save money and avoid this by instead buying a FIDO U2F Security Key. U2F is what Dropbox and Gmail use. I’m still working on figuring out how to not use text messages to my phone, since some of these services require both phone (which can be socially hacked from your provider, though that is well beyond normal phishing). One choice for some accounts is Google Authenticator (for a Mac); to use it requires physical access to the phone, not the account.

I would add, that at this point I feel a need to draw myself a graph of services and authentication methods and password managers (that can store data in the cloud on these services) to be sure that my access protection is “just right” — not so weak that it’s trivially hacked by phishing, not so strong that if I lose a single phone or key I am screwed.

Other stuff I use to help secure my laptop: Little Snitch (intercepts network connections) and Little Flocker (intercepts file I/O — i.e., ransomware). These tools are very annoying for normal people.

My attempts at explaining our net metering discussion, based on information from town meeting last night, discussions with both sides afterwards, and further discussions with friends who happen to know about some of the particulars of high-priced pump storage power suppliers.

I should add that there’s a huge issue with the framing of this issue, and it is possible to view the solar producers as overpaid electricity suppliers, or as freeloading users of our electrical grid, or as very aggressive energy conservers. I’m more interested in the town-wide question of our overall costs, our overall reliability, and our overall reduction in CO2 emissions.

What I did and didn’t figure out

This little section still a work in progress, but it is more or less a summary

The most important question is “where do we (want to) sit on the scale between patsies and freeloaders?” It appears that the bulk of the benefits from rooftop solar are spread either throughout the grid (reduced overall electric costs because of marginal-rate pricing) or throughout the world (tiny but non-zero reductions in CO2 emissions). A freeloader would advocate spending nothing, not because there are no benefits, but because we get the vast majority of the benefits from the contributions of patsies in other communities, so why shouldn’t we save our money? If we’re not willing to move some distance from “freeloader” there’s not too much point discussing the details of the benefits. The way the freeloader-vs-patsy problem is usually resolved is regulations and mandates from government — like the requirement on Investor-Owned-Utilities to do a certain amount of net-metering.

Another important thing is that whatever the benefits are, they are cumulative — we are not forced to look at the largest one and say “that’s not enough”. Rooftop solar reduces CO2 emissions AND reduces regional afternoon peak grid loads (thus reducing marginal cost) AND reduces hot-afternoon neighborhood loads (perhaps extending life of aging infrastructure, also reducing our consumption of very-high-priced regional-peak energy even though our peak is in the evening).

However, and this is also very important, there are alternatives to net metering that reduce loads more efficiently; for example, buying (good) LED light bulbs and giving them away for incandescent bulb replacement cuts a greater amount of energy consumption for the same amount of money, till we run out of bulbs to replace. LED light bulbs are not directly comparable because their savings aren’t as well-timed to regional peak loads as rooftop solar, but the overall energy savings are a good deal larger.

The benefits of net metering are likely of a decently large size (within a factor of two of the “subsidy”, one direction or the other) but very difficult to predict even in a given year. Effects on marginal cost pricing are often spiky, not smooth. Any effects on equipment lifetime are difficult to measure (it’s hard to observe the failure that didn’t happen). To the degree that they reduce peak loads, other conservation measures (LED lightbulbs, more efficient air conditioners, painting roofs white, improved roof circulation/insulation) also have a similarly spiky payoff that may over time yield substantial cost savings.

Yet more information, see also, “The Future of Solar Energy” (MIT). We probably get more bang for our buck if we spend our money on large-scale solar installations, rather than rooftop-by-rooftop. Details in the paper (I skimmed the executive summary), they make sense to me. Note one of the authors is Belmont resident and town meeting member Henry Jacoby.


Fixed costs and subsidies

The fixed portion of a typical electric bill is about 50%, but we don’t charge like that. That is, the usual price of electricity is designed to reward conservation and to be friendly to frugal users, who are disproportionately poor and/or retired. This is especially sensible in our current situation of tightly-constrained supply (non-Belmont readers – we have an aging bottleneck in our supply, and a replacement is under construction), but it remains sensible in the face of the need for CO2 emissions reduction.

Based on this crude calculation of fixed and variable costs, the 20 solar power producers in town who do net metering (who sell their excess production back to the light department and receive in return the full retail price for that energy) receive a “subsidy” of about $800 per year each, which results in higher costs to the non-solar ratepayers. The subsidy can either be viewed as underpayment of the fixed costs of a connection, or receiving far more for the energy they sell than other suppliers receive – either way, the light department says it’s $800. However, there’s 10,000 of us, and only 20 of them – per year, this subsidy costs each non-solar ratepayer about $1.60.

Compensating for social costs of CO2

Another way of looking at this number is to consider what we consider the social cost of a ton of CO2 – probably about $40. To put that into perspective, a $40 tax per CO2 ton is about 40 cents per gallon of fuel oil or gasoline (that is, burning 100 gallons of light petroleum yields about a ton of CO2). For power generation in the northeast, the “marginal emissions rate” is 914 lbs of CO2 per megawatt-hour, or 2200 kWh per ton of CO2. Marginal emissions rate is not the average over all production – it is the amount of CO2 emitted for the next MWh we request, or the amount avoided if we use 1 MWh less. (This calculation tends to underweight nuclear because the economics of nuclear plants tend towards always-on; the MWh  we avoid using will be something other than nuclear). As near as I can tell, each home solar installation avoids the consumption of about a MWh per year, perhaps double. From the pure social benefit view of things, we’ve declared that is worth $10 – $40 per year.  Each home solar installation produces on average about 6MWh per year, avoiding about 3 tons of CO2 emissions, and from a social benefit point of view that is worth about $120.  This would be double counting if we had a proper carbon tax, but we don’t yet. (Thanks to Roger Wrubel for this much better information.) One problem with accounting for the social benefit is that it is spread over many other people, not just the residents of Belmont — if we are expected to pay it, it should be required by regulation or imposed as a tax, and not voluntary. But contrary to that, note that right now it is a regulation that for-profit electric companies do net-metering; for those companies it is not voluntary.  Yet another complication; someone, somewhere, is probably getting a “Renewable Energy Credit” for that solar energy.  A friend in Texas with solar panels says that RECs are worth $60 per MWh, but he thinks the companies that do the “our panels on your roof” game have claimed them — and in some sense that is double-counting the social benefit.

HOWEVER, there are details that suggest that the story is not so simple or complete. There are (at least) three good reasons why solar is worth more than just the CO2-reduction benefit, or costs less than the claimed subsidy.

Need to calculate “subsidy” carefully

First, the price of electricity is not constant – it is more expensive during the day, and cheaper at night. Solar production only occurs during the day. The constant price charged by the light company is an average designed to let them break even considering consumption over 24 hours, 365 days per year. However, the average price is not too far below the usual daytime cost – the bulk of the electricity is consumed during the day, therefore the average price is going to be closer to the daytime cost. This is not that big a factor. But nonetheless, it is a factor, and it shaves a bit off the “subsidy” because solar energy is sold back during those hours when the power company’s supply costs are higher; in rare but somewhat predictable cases, the power company’s costs are so high that they are losing money on every additional kWh they sell to normal customers, and solar on the local net saves them money, even when they pay the full retail cost that they charge their customers.

Cutting (or not) the peak load.

The second reason has to do with cutting the peak load; there are (apparently) costs associated with wholesale power consumption and delivery that depend on Belmont’s peak power consumption over various periods of time. To the extent that solar reduces this peak, it saves money – however, for Belmont, the peak load usually occurs late in the afternoon or early in the evening well after the peak hours for solar production, so again, here the factor is not large. It would be nice to know what this formula is.

Marginal pricing and a silly market example

The third reason is the interesting one, and the tricky one. Why is electricity more expensive during the day? How is that price determined? The short, opaque answer is that the price of electricity is roughly determined by its marginal cost. Consider a simple example that will get more complex:

Suppose you had to make a 1000-egg omelette. For that many eggs, you pay attention to the cost of eggs, so you buy as many of the cheapest eggs you can find. Say, you can buy eggs at 8 cents each – but only 500, then that source runs out ($40 for those eggs). The next cheapest you can find cost 15 cents per egg, but you can only get 300 of those ($45 for those eggs), and you still need 200 more eggs. The next best price is $20 cents per egg, but that supplier only has 199 ($39.80 for those). The next cheapest supplier remaining always has eggs, but at a price of one dollar per egg (at those prices, it’s not surprising). Fortunately, you only need one. Your total cost is $125.80 for eggs.

According to one person I talked to, the suppliers in electric power markets are paid the marginal production cost, even if it is much higher than their bid. That means that if eggs were priced like power, on that day when you need to buy 1000 eggs and the last egg cost a dollar, you would pay a dollar for each of the eggs, even though 999 were offered at $.20 or less. The total cost is therefore $1000.

Another person I talked to says that’s not entirely right – that suppliers get paid what they bid, but the suppliers are not dummies. They know roughly how much capacity everyone has, they know the general plans for omelette-making, and they game their bids accordingly – so the bids were not for $.08, $.15, and $.20 – they were much higher, because they knew you were going to need a lot of eggs. It’s not quite the festival of price-fixing you might imagine it could be, because some suppliers (nuclear, in particular) cannot easily take their plants off-line, and they don’t save much money even if they could, there are many suppliers, and they are also a regulated utility.

If someone has better information on how this pricing works I would love to have it, and I would also be happy to align the egg prices to kWh supplier prices – or if it works better to ordinary egg prices, to make them a clean multiple, like double.

I found two better sources and they both suggest that the power market pricing really is based on marginal costs, with a few sensible exceptions for things like emergency reserves.  One paper is part of an ISO-NE slide deck on power pricing, and the other is a study of the effects of wind power on power markets in other parts of the country.

Note the importance of the marginal, last egg. Using power pricing rules, if you needed one less egg, then the last egg costs only $.20, and thus the cost of omelette supplies is only $199.80, and the savings are $800.20. Using kitchen pricing rules, that last egg avoided saves us only $1 – far less.

This looks a fabulous advantage for solar “eggs”, because they are usually available on summer afternoons when demand can be very high – but on a given day, you don’t know if you’re going to get lucky and avoid buying the eggs that make the difference between $1000 and $200. Some days your omelette is so big that you can’t avoid the expensive eggs, some days the low-cost suppliers have enough eggs that it’s not an issue, some days your omelette is small enough that it’s not an issue. The demand and supply have to be “just right” to get lucky like this. But when it happens, it’s a really big win. Does it happen often? Is the win that big?  I don’t know any of this for sure.

Who gets the benefits of our solar “spending” / are we freeloading on the IOUs?

There is a further caveat; the price of electricity is set regionally in a regional market based on regional supply and demand. When the highest-priced supplier’s energy is not needed, everybody wins. A solar panel in Arlingon is just as effective at this as a solar panel in Belmont. Therefore, a Belmont electric company that is only interested in what is best for Belmont might quite reasonably choose to let other communities install solar, and we will benefit from that at no cost to Belmont. Or, alternately the electric company might feel, and again quite reasonably, that if they are going to take $16,000 from the pockets of their rate-payers each year, that it might better be spent buying some other peak-shaving energy supply where we obtain more load reduction per dollar. This would be even more reasonable if the electric company could point to where they had spent $16,000 and obtained a more effective reduction in demand or CO2 emissions.

We pay for insurance, we pay for reliable reserve capacity

And on top of that, I think there is yet more to worry about. We need a certain supply of responsive, expensive backup generation “just in case”, and it’s no surprise that sometimes it has fixed costs, and those fixed costs are only paid when they can sell energy into expensive peak markets. Their fixed costs don’t go away just because there’s less demand for their services, and if those costs have to be amortized over less use, then either the cost of peak-load generation goes up or else the peak-load generators go out of business. Solar might let us pay for peak supply less often, but at those times when we still need, it might be more expensive. There’s a limit to how high these costs can rise; if/when it is cheaper to install large batteries around the electrical grid, we’ll do that instead – and that cost will come down over time, too.
The wind power study linked above mentions this also — wind is variable, therefore some quick-starting reserves need to be kept on hand, and these have expenses that cannot be met in a normal power market:

“Although fundamentally very different, both MISO and PJM operate a capacity market that is designed to provide a source of income to generators that may not sell enough electricity to be economically viable, but are necessary to RTOs in order to satisfy target reserve margins and ensure system reliability. As more variable energy sources are added to an RTO system, the premium for reserve capacity could rise.”

Delivery also has costs, not sure how they are priced.

The eggs and omelettes example can be extended to include the problems of electrical delivery. Here in Belmont we are constrained by the limits of our aging substation, and that is why we are spending money to replace it. But this problem occurs in general throughout the grid; wires have capacities, substations have capacities, nothing is perfectly efficient, and sometimes it is broken. It as if eggs were delivered by bicycle messengers – some ride big bikes, some ride little bikes, you have to be sure that you have enough messengers to deliver your eggs when you need them, even though other omelette-makers are bidding for their services at the same time. And, sometimes the messengers hit bumps and break a few eggs, and the further the eggs must travel the more bumps they hit and the more eggs you lose. In rare cases, a messenger crashes, all the eggs they were carrying are lost, and the messenger and/or bicycle is out of commission for a while. When there aren’t enough low-priced messengers to get the eggs that you need when you need them, then you need to pay more.

Here, I have not yet heard or figured out what the pricing structure for electricity/egg delivery is – because delivery is less interchangeable than electrical energy, it doesn’t make sense for everyone to pay the cost of the most expensive link in the system. So I don’t know exactly what is going on in this market except that I am sure that prices will spike when links are near their capacity. I know that when there is only one messenger service that everyone must contract with, they can game the prices to make a lot of money – that was Enron’s business.

Alternatives?

I have some sympathy for the light company’s position, but not at all for their explanation. It’s not as simple as “we subsidize them”, nor is the “subsidy” as large as is claimed. I’m not exactly begrudging the current home solar generators the $1.60 that I toss their direction each year, and it’s a small enough amount of money that I’d like to see the light company spending that much each year installing or investing in a better source of renewable, peak-shaving power, subsidizing conservation, or something similar. It would prove their point very effectively.

It need not be solar – a good-sized battery at our new substation might allow us to smooth the peaks of our consumption, completely avoid last-minute purchases, and perhaps even to sell into that market. That might not make sense at today’s battery and electricity prices, but those are both changing.

Below, an earlier version of this contained a math error — LED bulbs are indeed a very superior investment.

Or consider what $16,000 spent on LED light bulbs to replace incandescent bulbs will return in one year, and over time. At current prices that will buy 640 (good) “100-watt replacement” bulbs, and each of those bulbs uses 80 watts less. If we assume that 640 bulbs are each operated 5 hours each day, that is 256kWh saved per day, times 365 days gives you a little over 90MWh savings per year. Each year we buy more bulbs and save more, till we run out of incandescent light bulbs to replace. If half the homes in town have only one such costly light bulb in common use, it would still take us 8 years to do this. However, even after eight years of spending at this level we’ve only cut our annual consumption by 8MWh, versus the (estimated) 120MWh that rooftop solar saves us. For one year, rooftop solar is ahead, but after two years of LED purchase the savings per year are 180MWh, and after 5 years 450MWh — 3.75x our savings from solar. Notice that solar is better timed to the regional peak, but LED lightbulbs are better timed to the Belmont (early-evening) peak, and also have the effect of reducing the A/C load on summer evenings by a small but non-zero amount.

So LED light bulbs are generally a better way for the light department to spend money for load reduction, till we have run out of places to put them. (Note that in a town full of rational consumers, we would already have LED light bulbs in all sockets run more than two hours per day — over a period of 5 years, such a light bulbs saves $11.50 per year, and nobody would pass up that deal just because the light department had given away their annual quota of light bulbs).

(Note also that if you worry about losing heat from light bulbs in the winter, that resistive heating is not a great source of energy – with a heat pump you get 3 times as much warmth for a given amount of energy consumption, and you get it where you want it, not close to the ceiling warming up a room you’re not in or the snow on your roof).

Energy Auctions

After reading more papers and thinking about how the auctions work, I think the rule for marginal pricing is that it applies to each auction, and there are multiple auctions. There’s an auction the day before; there’s an hourly auction the next day, and I think there is an auction every five minutes. The marginal price in the day-before auction determines what is paid in that auction; the marginal price in the hourly auction determines what is paid in that auction, and similarly for the five minute auctions. But the marginal prices in the five minute auction (which may be very high) I think only apply to that auction — it does not result in more money in the pockets of the suppliers who sold into the previous evening’s auction, or in the most recent hourly auction. And because it is for only a short amount of time it is not as much overall power as what is bid for in the hourly markets, hence the high price for energy only applies to a smallish amount of energy (usually, you hope).

An interesting thing to notice is that wind appears to sell only into the short-term market of at least one RTO (regional transmission operator PJM, mid-Atlantic plus Ohio and a bit more). This makes a certain amount of sense; we have a better idea how hard the wind will be blowing just an hour ahead of time, rather than 12-24 hours ahead of time, and the prices tend to be higher in those markets — and the wind providers bid $0 so they are guaranteed to get in on whatever market there is, knowing that they will be paid marginal price, not their bid:
“During calendar year 2011, wind represented 2% of the marginal generation used for the real-time energy market. Wind was never a marginal generation source for the day-ahead market in 2011. As discussed above, the day-ahead and real-time markets typically include 95% and 5% of energy transactions, respectively. In 2011, wind power accounted for 1.5% of electricity generation in PJM.”
This has the potentially unfortunate effect of putting wind more directly into competition with short-term reserve supplies. Some RTOs establish separate markets for “capacity” that tries to capture the ability to reliably deliver electricity on demand, and wind does not score well on this metric.

And further:

In 2009, PJM conducted a study that considered the wholesale power price impacts of adding 15,000 MW of wind power in the PJM market. Results from the study indicated that the addition of wind power would decrease wholesale market prices by $4.50 per MWh. As a result, market-wide expenditures for wholesale power would go down. For comparison, PJM’s system-wide load-weighted average LMP was $45.19 in 2011.

Adding 15,000 MW of wind power would be a quadrupling, from about 2.5% of PJM’s total generation capacity to about 10% of total generation capacity. Here, it’s not clear if “wholesale market prices” mean over the entire power generation market (day-ahead and real-time) or only in the smaller real-time market.

I am not much closer to knowing how much a given amount of rooftop solar reduces the marginal cost of electricity. One particular difficulty in comparing effects from this paper is that wind tends to peak when demand is lower, the opposite of solar:

“The profile of wind generation is inversely correlated with the load demand profile in ERCOT. Much like other regions of the country, when load demand is high, wind production is low and vice versa.”

That means solar ought to have a relatively larger effect in reducing the wholesale market price (for the definition used in that paper).

Here’s an interesting source of information that I have yet to digest. I am trying to figure out what fraction of the total ISO-NE capacity solar represents, and how much an additional fraction of solar is likely to affect the overall wholesale price (in what market, for how much energy? day ahead? real time?) and to compare that to Belmont’s contribution. And by “figure”, I mean an educated conservative guess — if moving from 2.5% to 10% wind in PJM cuts wholesale prices by 10%, we might expect a similar inverse relation (or better, because of better alignment with regional peak demand) in the 0-10% range for solar. Useful/interesting facts so far — Massachusetts “nameplate” solar capacity is currently 666MW, forecast to roughly double by 2023 or 2024. ISO-NE’s solar is 909, forecast to rise to 2450MW through 2024. This document from ISO-NE suggests actual delivered solar power of 331 GWH out of a total of 127,108 GWH for 2014, or about 1/4 of one percent. It would not be outlandish to project savings of a similar magnitude or larger in “wholesale prices” — the effect is diminishing with larger scale, 0.25% is not much scale, PJM sees a 10% reduction in “wholesale prices” by increasing wind’s share to 10%, and wind isn’t even well-timed against high load like solar is. Note that wholesale prices are only a fraction of retail prices, but based on the figures here I estimate that the claimed solar subsidy costs each Belmont customer not quite 0.13% of their retail bill.

Feedback and questions

So, enough details? Constructive comments and corrections are extremely welcome, I’ve got no time for ad hominem attacks, imputing nefarious motives, and I take a dim view of cherry-picking information. Checking my math is great; I’ve found one large error, though the rest of the numbers seem about right. I’d love references. I’d love better explanations of how the electricity market works (I’ve been working hard to figure this out, there are apparently rounds of bidding that occur a day ahead, an hour ahead, and then every five minutes, and apparently you really want to avoid spending money in the five-minutes market. “The rugby team just showed up, I need to make a ginormous omelette, pronto! What, we’re out of eggs?”)

A question via email from friend JF: “Right now, the cost of the solar panel pay back (since subsidy doesn’t seem to be the right word) for the majority of us is rather low. But won’t that increase as users increase? In that case, won’t what it costs non-solar users like me get to be something that isn’t acceptable? (I do understand that this will mean that the number of solar users will increase and the number non-solar users will decrease, of course. )”

The answer is yes, certainly. On the other hand if it turns out that there is a net benefit to rooftop solar, then for small multiples of the current number of installations the benefits also increase. There is a point at which the increase in benefits falls off, but I think this would not happen even for ten times as many rooftop solar installations (i.e., 200, out of 10000, or 2%). Note again that we don’t get to capture the full magnitude of the benefits, and nothing prevents us from capturing the benefits of other solar installations — there is an incentive to (in game thoretic terms) “freeload”.

One point made in conversation with the light company is that it is important to “get the price right”. I think they are worried about a range of possible futures, that might include much more affordable solar, a rapid increase in the number of installations, and a widespread expectation of net metering prices for solar (or worse, a lot of people who had committed money to a solar installation and perhaps now have a strong financial interest in continuing it).

Questions (not a complete list)

Who’s getting the RECs for rooftop solar in Belmont?

Are we already paying something like a carbon tax for some or all of our power?
I think we are — $10 per ton — but I’m not sure of the details.

In what ways is Belmont Light already spending money to reduce electrical demand, peak or otherwise? How effective is that spending? (Need to start here to look for answers.)

Are we being clear about what we think our social obligations are? Do we wish to err in the direction of being freeloaders (doing less, getting more) or chumps (doing more, getting less)?

Related interesting stuff

An interesting summary of the economics of the Tesla PowerWall. Note their uncertainty about how the cost of the inverter is accounted in SolarCity’s proposed package deal; I’m glad to know I’m not the only puzzled person here.

Notes on I2C on Atmega328p

February 7, 2015

Registers:

TWBR = bit rate

TWCR =
  TWINT interrupt flag
  TWEA enable acknowledge
  TWSTA set start
  TWSTO set stop
  TWWC collision detected
  TWEN twi enable
  – reserved
  TWIE interrupt enable

TWSR =
  TWS(7:3) status
  – reserved
  TWPS (1:0) prescaler
 

TWDR = data register

SCL freq = CPU clock / (16 + 2 * TWBR * 4TWPS).

TWBR must be at least 10 for master mode.

SCL freq must be 100Khz or less.
CPU clock is 8 Mhz.
Denominator must be larger than 80 = 16 + 64, TWBR * 4TWPS must be greater than or equal to 32.
TWBR=10-31, TWPS=1 (40-124)
TWBR=32-255, TWPS=0 (32-255)
At 100kHz, send data at about 10kB/s. 80 characters requires 8ms.

Master-side state changes for writing a data byte to a slave:

TWCR = _BV(TWSTA)|_BV(TWINT)|_BV(TWEN)|_BV(TWIE)
Receive interrupt, check TWSR for successful start.

TWDR = SLA+W (W=0, R=1: SLA = 0x50 for Newhaven serial display
TWCR = _BV(TWINT)|_BV(TWEN)|_BV(TWIE)
Receive interrupt, check TWSR for successful address.

TWDR = data
TWCR = _BV(TWINT)|_BV(TWEN)|_BV(TWIE)
Receive interrupt, check TWSR for successful data.

repeat data.

TWCR = _BV(TWSTO)|_BV(TWINT)|_BV(TWEN)|_BV(TWIE)
Receive interrupt, check TWSR for successful stop. Read the rest of this entry »

Running a self-driving “car” back-and-forth across a street at low speed ought to be easy compared to the general problem of driving, and it would serve a public good.

We can’t afford to staff every intersection with its own human crossing guard, but places like New York City someone in a crosswalk gets hit and killed every week, if not more so.

Rather than expect the NYPD “No criminality suspected” culture to change, let’s use robotics to even the playing field. Whenever a pedestrian should have right of way, a robot crossing guard (or two) accompanies them across the street, between them and potentially oncoming traffic. The crossing guard would be tall, built very sturdily, and weigh about, oh, 5 tons, with most of that carried low so it won’t tip over in an impact. That will give drivers an incentive to pay a little more attention to what is in front of them, and also protect pedestrians when they don’t.

I tested DHCP Client on Mac OSX Mavericks to see if it is vulnerable to the Bash hole. If my test is correct, it is not. Here is my test:

First, come up with a root command that will be noticeable, and verify that it is noticeable. /usr/bin/wall is one such command, and I tested it here:

dr2chase:VM dr2chase$ sudo /usr/bin/wall /etc/syslog.conf
Password:
                                                                               
Broadcast Message from dr2chase@dr2chase.local                                 
        (/dev/ttys001) at 22:03 EDT...                                         
                                                                               
# Note that flat file logs are now configured in /etc/asl.conf                 
                                                                               
install.*						@127.0.0.1:32376                                                
                                                                               

Next, open a window to my router running Tomato and feed options to Dnsmasq, save, wait for the services to restart, then turn wifi off and on.

DnsMasqSettings

For copy-paste purposes, that string is

dhcp-option-force=114,() {ignored;}; /usr/bin/wall /etc/syslog.conf

I also tried this string to see if the bash script was running with lower privileges, yet still vulnerable:

dhcp-option-force=114,() {ignored;}; /bin/cp /etc/syslog.conf /tmp

I used these examples to get the option-setting right to Tomato, and this example to get the right option string for dnsmasq.

I verified this by setting log-dhcp and checking the logs on the router, and saw this:

Sep 27 22:25:33 janus daemon.info dnsmasq-dhcp[4580]: 1981429455 sent size: 45 option:114   28:29:20:7b:69:67:6e:6f:72:65:64:3b:7d:3b...

It should be intuitively obvious to the ASCII observer that the string was sent, spaces and all.