It's a bit surprising where the difficulties of Amateur ATSC TV lie.
It turns out that getting MPEG encoding gear isn't going to be very difficult after all. Part of this is due to the work of European hams, who have been transmitting digital TV using European standards for years now. North American ATSC standards, however, are more challenging.
At first I thought that I'd be able to simply take the output of the MPEG encoder and feed it into the 8VSB modulator, but it appears that the result, though it would be a valid MPEG 2 transport stream, would not be good enough for ATSC. There are three issues remaining that I've identified:
1. The audio has to be Dolby AC3. ATSC doesn't support any other audio format, and AC3 is heavily patent encumbered, meaning there are no low cost encoders for it. You can encode it with open source software, but now you're using a PC as your encoder, with all of the complications and reduced reliability that entails.
2. ATSC requires a constant 19.39 MB/s data rate. You're supposed to stuff the channel with null packets to make it so if your data is less than that.
3. The dreaded PSIP. PSIP is a complete waste of time and effort for amateur TV, but if you want the thing to decode properly with consumer grade tuners or receivers, you've got to generate at least a static set of PSIP tables.
On top of all of that, one other ham I've talked to who is extremely experienced with ATV says that 900 MHz and 1.2 GHz in the bay area are more or less useless due to heavy interference. And if that weren't enough, there's 8VSB's sensitivity to non-linearity in the RF chain, which has implications for the design of your power amplifier.
European hams have had success with DVB-T, which doesn't have these issues, but you can't use ordinary North American consumer equipment to decode it, which makes it (at least here) just another specialized ham-only mode. The value of attempting to transmit ATSC compatible video is that just adding a downconverter to a standard consumer system is the only change required, which expands the potential receiver pool exponentially.
This is looking less possible daily. :(